Nov 02 07:21:35.248284 ip-10-0-135-6 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 07:21:35.248299 ip-10-0-135-6 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 07:21:35.248308 ip-10-0-135-6 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 07:21:35.248664 ip-10-0-135-6 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 07:21:45.435656 ip-10-0-135-6 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 07:21:45.435672 ip-10-0-135-6 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 857520d6fcfc4003aef278b6f591d457 -- Nov 02 07:23:47.732427 ip-10-0-135-6 systemd[1]: Starting Kubernetes Kubelet... Nov 02 07:23:48.129301 ip-10-0-135-6 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 07:23:48.129301 ip-10-0-135-6 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 07:23:48.129301 ip-10-0-135-6 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 07:23:48.129301 ip-10-0-135-6 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 07:23:48.132345 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132269 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:23:48.132345 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132346 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132350 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132355 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132361 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132365 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132368 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132371 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132374 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132377 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132380 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132382 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132385 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132387 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132390 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132396 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132399 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132402 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132405 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132407 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:23:48.132418 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132410 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132413 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132416 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132418 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132421 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132424 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132427 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132429 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132433 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132435 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132438 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132441 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132443 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132446 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132448 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132451 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132453 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132456 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132458 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:23:48.132854 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132462 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132466 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132469 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132472 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132474 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132477 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132480 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132482 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132485 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132487 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132490 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132493 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132495 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132499 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132502 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132504 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132507 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132510 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132513 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132516 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:23:48.133322 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132518 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132521 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132524 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132526 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132529 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132531 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132533 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132536 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132538 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132541 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132543 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132546 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132548 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132551 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132554 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132556 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132559 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132561 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132564 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132566 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:23:48.133789 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132569 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132571 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132574 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132577 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132579 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132583 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132586 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132589 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132591 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132594 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132597 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132599 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132602 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132604 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132607 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132609 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132612 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132614 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132616 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:23:48.134278 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132619 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132622 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132624 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132627 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132631 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132633 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132636 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.132639 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133418 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133425 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133429 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133432 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133435 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133437 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133440 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133443 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133445 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133448 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133450 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:23:48.134748 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133453 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133457 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133459 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133462 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133464 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133467 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133470 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133473 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133475 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133478 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133480 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133483 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133486 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133488 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133491 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133494 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133496 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133499 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133501 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133504 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:23:48.135218 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133506 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133509 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133512 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133515 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133517 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133520 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133523 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133525 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133530 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133534 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133537 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133541 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133543 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133546 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133549 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133552 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133554 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133557 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133559 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133562 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:23:48.135697 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133565 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133567 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133570 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133573 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133575 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133578 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133580 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133583 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133585 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133588 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133590 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133593 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133595 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133598 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133601 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133604 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133606 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133609 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133611 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:23:48.136179 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133614 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133616 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133619 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133622 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133624 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133627 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133630 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133632 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133635 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133637 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133641 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133645 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133648 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133652 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133655 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133657 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133660 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133663 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133666 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133668 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:23:48.136629 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133671 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133674 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133676 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133679 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133681 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133684 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133686 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133690 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133693 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133695 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133698 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133701 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133703 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133706 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133709 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.133711 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134239 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134246 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134250 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134253 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:23:48.137092 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134256 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134260 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134263 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134266 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134269 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134271 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134274 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134277 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134280 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134283 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134285 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134288 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134291 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134294 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134297 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134299 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134303 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134307 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134310 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:23:48.137566 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134313 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134317 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134319 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134322 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134325 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134328 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134333 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134336 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134339 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134342 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134344 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134347 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134349 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134352 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134354 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134357 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134359 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134362 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134364 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:23:48.138012 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134367 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134370 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134372 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134374 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134377 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134379 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134382 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134384 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134387 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134389 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134392 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134394 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134397 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134399 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134402 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134405 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134408 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134410 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134413 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:23:48.138477 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134415 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134418 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134421 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134424 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134426 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134429 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134431 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134434 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134436 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134439 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134442 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134444 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134447 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134449 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134452 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134455 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134457 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134459 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134462 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134464 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:23:48.138927 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134467 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134469 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134472 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134474 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134477 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134479 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134482 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134484 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134487 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134490 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134492 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134495 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134497 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134500 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134502 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134505 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134508 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134511 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134513 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134516 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:23:48.139456 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134518 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134520 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134524 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134526 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.134529 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134613 2577 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134620 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134627 2577 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134631 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134636 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134639 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134643 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134647 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134650 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134653 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134657 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134661 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134664 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134667 2577 flags.go:64] FLAG: --cgroup-root="" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134670 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134673 2577 flags.go:64] FLAG: --client-ca-file="" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134676 2577 flags.go:64] FLAG: --cloud-provider="external" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134679 2577 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134683 2577 flags.go:64] FLAG: --cluster-domain="" Nov 02 07:23:48.139923 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134686 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134689 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134692 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134696 2577 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134702 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134705 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134708 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134711 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134714 2577 flags.go:64] FLAG: --contention-profiling="false" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134718 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134721 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134724 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134727 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134731 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134734 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134737 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134740 2577 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134743 2577 flags.go:64] FLAG: --enable-server="true" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134746 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134751 2577 flags.go:64] FLAG: --event-burst="100" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134754 2577 flags.go:64] FLAG: --event-qps="50" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134757 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134760 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134763 2577 flags.go:64] FLAG: --eviction-hard="" Nov 02 07:23:48.140490 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134768 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134771 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134774 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134777 2577 flags.go:64] FLAG: --eviction-soft="" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134780 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134783 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134786 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134789 2577 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134793 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134796 2577 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134799 2577 flags.go:64] FLAG: --feature-gates="" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134803 2577 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134808 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134811 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134814 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134817 2577 flags.go:64] FLAG: --healthz-port="10248" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134820 2577 flags.go:64] FLAG: --help="false" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134823 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134826 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134830 2577 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134832 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134836 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134839 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134842 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 07:23:48.141212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134845 2577 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134848 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134851 2577 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134853 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134856 2577 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134859 2577 flags.go:64] FLAG: --kube-reserved="" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134862 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134865 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134868 2577 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134870 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134873 2577 flags.go:64] FLAG: --lock-file="" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134876 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134879 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134882 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134888 2577 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134893 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134897 2577 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134900 2577 flags.go:64] FLAG: --logging-format="text" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134903 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134906 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134911 2577 flags.go:64] FLAG: --manifest-url="" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134914 2577 flags.go:64] FLAG: --manifest-url-header="" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134918 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134921 2577 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134925 2577 flags.go:64] FLAG: --max-pods="110" Nov 02 07:23:48.141774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134928 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134931 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134934 2577 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134937 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134940 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134943 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134946 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134952 2577 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134956 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134958 2577 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134962 2577 flags.go:64] FLAG: --pod-cidr="" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134965 2577 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134968 2577 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134971 2577 flags.go:64] FLAG: --pods-per-core="0" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134975 2577 flags.go:64] FLAG: --port="10250" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134978 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134981 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-097f5d70939a7d82e" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134984 2577 flags.go:64] FLAG: --qos-reserved="" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134987 2577 flags.go:64] FLAG: --read-only-port="10255" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134990 2577 flags.go:64] FLAG: --register-node="true" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134993 2577 flags.go:64] FLAG: --register-with-taints="" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.134997 2577 flags.go:64] FLAG: --registry-burst="10" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135001 2577 flags.go:64] FLAG: --registry-qps="5" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135004 2577 flags.go:64] FLAG: --reserved-cpus="" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135007 2577 flags.go:64] FLAG: --reserved-memory="" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135011 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 07:23:48.142388 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135014 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135019 2577 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135022 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135025 2577 flags.go:64] FLAG: --runonce="false" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135028 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135031 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135034 2577 flags.go:64] FLAG: --seccomp-default="false" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135037 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135040 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135043 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135046 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135049 2577 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135052 2577 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135055 2577 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135058 2577 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135061 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135064 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135067 2577 flags.go:64] FLAG: --system-cgroups="" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135070 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135075 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135078 2577 flags.go:64] FLAG: --tls-cert-file="" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135080 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135084 2577 flags.go:64] FLAG: --tls-min-version="" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135087 2577 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135090 2577 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 07:23:48.143027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135093 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135096 2577 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135099 2577 flags.go:64] FLAG: --v="2" Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135104 2577 flags.go:64] FLAG: --version="false" Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135108 2577 flags.go:64] FLAG: --vmodule="" Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135112 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135116 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135293 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135302 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135305 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135308 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135311 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135314 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135317 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135319 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135322 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135325 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135327 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135330 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135333 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135335 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:23:48.143649 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135338 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135340 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135343 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135345 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135348 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135351 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135353 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135356 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135358 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135361 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135364 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135366 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135369 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135371 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135375 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135378 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135380 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135384 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135389 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135393 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:23:48.144132 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135396 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135399 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135402 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135404 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135409 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135412 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135415 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135418 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135420 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135423 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135425 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135428 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135431 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135433 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135436 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135439 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135441 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135444 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135446 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:23:48.144618 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135449 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135452 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135454 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135457 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135459 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135462 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135464 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135468 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135470 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135473 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135475 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135478 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135489 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135492 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135494 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135497 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135499 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135502 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135505 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135507 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:23:48.145074 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135510 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135513 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135515 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135518 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135520 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135522 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135525 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135528 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135530 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135533 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135535 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135538 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135540 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135542 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135545 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135547 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135550 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135552 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135555 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:23:48.145547 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135559 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135561 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135564 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135567 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135569 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135573 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135581 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135584 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135587 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135589 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135592 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135594 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135597 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135600 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.135626 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135669 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:23:48.145978 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135673 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135676 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135679 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135682 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135684 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135687 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135689 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135692 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135694 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135697 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135701 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135705 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135708 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135711 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135713 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135716 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135718 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135724 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135728 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:23:48.146363 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135730 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135734 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135737 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135740 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135748 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135752 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135755 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135758 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135760 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135763 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135766 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135768 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135771 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135773 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135776 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135778 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135781 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135783 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135786 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135788 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:23:48.146810 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135791 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135793 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135796 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135798 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135800 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135803 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135806 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135808 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135811 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135813 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135817 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135820 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135822 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135824 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135827 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135830 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135832 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135842 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135845 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:23:48.147427 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135847 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135850 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135852 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135855 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135857 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135860 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135862 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135865 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135867 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135870 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135872 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135875 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135877 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135880 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135882 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135885 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135887 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135890 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135892 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:23:48.148107 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135895 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135897 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135900 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135902 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135907 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135910 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135912 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135915 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135918 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135927 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135929 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135933 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135941 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135944 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135947 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135949 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135952 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135954 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135957 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:23:48.148663 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135959 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135962 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135964 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135967 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135970 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135972 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135975 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135977 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.135980 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.142852 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.142866 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.143603 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.143610 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143684 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143689 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143692 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143694 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143698 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143702 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:23:48.149138 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143706 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143709 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143712 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143714 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143717 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143720 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143722 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143725 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143727 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143730 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143732 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143735 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143738 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143740 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143743 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143745 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143748 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143750 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143753 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143755 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:23:48.149609 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143758 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143761 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143764 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143774 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143777 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143780 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143782 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143785 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143788 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143790 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143792 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143795 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143798 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143800 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143803 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143806 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143808 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143810 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143813 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143816 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:23:48.150077 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143818 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143821 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143823 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143826 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143828 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143830 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143833 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143835 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143838 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143841 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143843 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143846 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143849 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143851 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143853 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143856 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143865 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143868 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143871 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:23:48.150613 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143873 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143876 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143879 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143881 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143884 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143888 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143892 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143895 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143898 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143901 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143904 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143906 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143909 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143911 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143914 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143916 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143919 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143921 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143924 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:23:48.151050 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143926 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143929 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143931 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143939 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143942 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143945 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143947 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143950 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143952 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143955 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143957 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143966 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143969 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143971 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143973 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143976 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143979 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143981 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143983 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143986 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:23:48.151516 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143988 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.143991 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.144019 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144243 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144247 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144251 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144254 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144256 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144259 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144261 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144264 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144267 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144270 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144272 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144274 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144277 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:23:48.151977 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144279 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144282 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144284 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144286 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144289 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144291 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144295 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144299 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144308 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144311 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144314 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144318 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144322 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144325 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144328 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144331 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144333 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144335 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144338 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:23:48.152472 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144340 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144343 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144345 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144348 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144351 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144353 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144355 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144358 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144360 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144363 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144365 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144367 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144370 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144372 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144375 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144377 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144380 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144383 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144385 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144388 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:23:48.152906 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144390 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144393 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144401 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144404 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144407 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144409 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144411 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144414 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144416 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144419 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144421 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144424 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144426 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144429 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144431 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144433 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144436 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144438 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144441 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144443 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:23:48.153381 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144449 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144452 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144454 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144457 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144459 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144462 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144464 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144466 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144469 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144472 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144474 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144477 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144481 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144484 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144487 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144496 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144499 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144501 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144504 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144507 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:23:48.153834 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144509 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144512 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144514 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144517 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144519 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144522 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144524 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144527 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144529 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144532 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144534 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144537 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144540 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144542 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.144565 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144604 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:23:48.154326 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144608 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144611 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144614 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144617 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144620 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144623 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144625 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144628 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144631 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144635 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144638 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.144641 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145018 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145021 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145024 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145027 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145029 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145032 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145034 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:23:48.154702 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145037 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145039 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145042 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145045 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145047 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145050 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145053 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145055 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145058 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145060 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145063 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145065 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145068 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145071 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145073 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145076 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145079 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145081 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145084 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145087 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:23:48.155182 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145089 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145092 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145096 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145099 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145102 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145104 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145108 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145110 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145113 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145116 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145118 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145121 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145124 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145126 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145128 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145131 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145134 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145136 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145139 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145141 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:23:48.155658 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145144 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145146 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145149 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145151 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145154 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145167 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145170 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145173 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145175 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145178 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145180 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145183 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145185 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145188 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145192 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145195 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145197 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145200 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145204 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145207 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:23:48.156123 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145209 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145214 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145216 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145219 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145221 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145224 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145226 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145229 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145233 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145237 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145240 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145242 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145245 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145249 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145253 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145255 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145258 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145261 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:23:48.156627 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145264 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145266 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145269 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145271 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145274 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145277 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145279 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:48.145283 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.145339 2577 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.150055 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.150975 2577 server.go:1014] "Starting client certificate rotation" Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.151065 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.151104 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.153427 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 07:23:48.157030 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.154928 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 07:23:48.165905 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.165888 2577 log.go:25] "Validated CRI v1 runtime API" Nov 02 07:23:48.168756 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.168741 2577 log.go:25] "Validated CRI v1 image API" Nov 02 07:23:48.169635 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.169607 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 07:23:48.176242 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.176223 2577 fs.go:135] Filesystem UUIDs: map[53eee952-6703-41c0-a437-6a506ab1bf7a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f3a16f42-222b-4922-bac7-4ae558733e5c:/dev/nvme0n1p4] Nov 02 07:23:48.176313 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.176241 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 07:23:48.182444 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.182331 2577 manager.go:217] Machine: {Timestamp:2026-11-02 07:23:48.180400744 +0000 UTC m=+0.333034012 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094851 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21640333e3786bc122adcd83e39fca SystemUUID:ec216403-33e3-786b-c122-adcd83e39fca BootID:857520d6-fcfc-4003-aef2-78b6f591d457 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f2:b3:d6:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f2:b3:d6:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:11:f5:fc:e0:61 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 07:23:48.182444 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.182441 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 07:23:48.182623 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.182543 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 07:23:48.185872 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.185850 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 07:23:48.186011 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.185874 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-6.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 07:23:48.186060 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.186014 2577 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 07:23:48.186060 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.186022 2577 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 07:23:48.186060 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.186047 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 07:23:48.186060 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.186059 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 07:23:48.186880 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.186870 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 07:23:48.190837 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.190825 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 07:23:48.190960 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.190951 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 07:23:48.191610 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.191600 2577 kubelet.go:493] "Attempting to sync node with API server" Nov 02 07:23:48.191647 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.191618 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 07:23:48.191647 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.191636 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 07:23:48.191647 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.191646 2577 kubelet.go:405] "Adding apiserver pod source" Nov 02 07:23:48.191764 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.191653 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 07:23:48.191973 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.191962 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 07:23:48.195931 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.195903 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 07:23:48.200766 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.200753 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 07:23:48.201555 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.201542 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 07:23:48.201593 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.201573 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 07:23:48.201722 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.201712 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 07:23:48.201769 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.201739 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 07:23:48.201769 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.201750 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 07:23:48.201769 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.201758 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 07:23:48.201769 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.201766 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 07:23:48.201896 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.201774 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 07:23:48.202532 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.202523 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 07:23:48.202562 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.202533 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 07:23:48.202562 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.202541 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 07:23:48.202562 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.202547 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 07:23:48.202562 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.202555 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 07:23:48.202676 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.202564 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 07:23:48.203891 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.203880 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 07:23:48.203891 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.203891 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 07:23:48.207700 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.207687 2577 server.go:1263] "Started kubelet" Nov 02 07:23:48.207794 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.207765 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 07:23:48.207843 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.207788 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 07:23:48.207887 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.207863 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 07:23:48.208469 ip-10-0-135-6 systemd[1]: Started Kubernetes Kubelet. Nov 02 07:23:48.209015 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.208998 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 07:23:48.212398 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.210546 2577 server.go:317] "Adding debug handlers to kubelet server" Nov 02 07:23:48.214411 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.214380 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 07:23:48.214612 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.214589 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 07:23:48.214694 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.214590 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 07:23:48.214694 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.214688 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:23:48.215114 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.215099 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 07:23:48.215501 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.215483 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 07:23:48.215501 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.215503 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 07:23:48.215631 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.215581 2577 reconstruct.go:97] "Volume reconstruction finished" Nov 02 07:23:48.215631 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.215590 2577 reconciler.go:29] "Reconciler: start to sync state" Nov 02 07:23:48.215970 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.215958 2577 factory.go:153] Registering CRI-O factory Nov 02 07:23:48.215970 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.215972 2577 factory.go:223] Registration of the crio container factory successfully Nov 02 07:23:48.216047 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.216025 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 07:23:48.216047 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.216032 2577 factory.go:55] Registering systemd factory Nov 02 07:23:48.216047 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.216037 2577 factory.go:223] Registration of the systemd container factory successfully Nov 02 07:23:48.216130 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.216049 2577 factory.go:103] Registering Raw factory Nov 02 07:23:48.216130 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.216058 2577 manager.go:1199] Started watching for new ooms in manager Nov 02 07:23:48.216935 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.216922 2577 manager.go:319] Starting recovery of all containers Nov 02 07:23:48.228850 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.228816 2577 manager.go:324] Recovery completed Nov 02 07:23:48.233844 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.233831 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 07:23:48.235582 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.235557 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientMemory" Nov 02 07:23:48.235656 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.235594 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 07:23:48.235656 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.235604 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientPID" Nov 02 07:23:48.236080 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.236066 2577 cpu_manager.go:226] "Starting" policy="none" Nov 02 07:23:48.236133 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.236078 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 07:23:48.236185 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.236080 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 07:23:48.236225 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.236191 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 07:23:48.236729 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.236715 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientMemory" Nov 02 07:23:48.236779 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.236740 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 07:23:48.236779 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.236750 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientPID" Nov 02 07:23:48.238385 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.238374 2577 policy_none.go:50] "Start" Nov 02 07:23:48.238428 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.238388 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 07:23:48.238428 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.238398 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 07:23:48.240474 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.240461 2577 policy_none.go:44] "Start" Nov 02 07:23:48.244526 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.244499 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 07:23:48.244526 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.244504 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-6.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 07:23:48.245652 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.244506 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-6.ec2.internal.18e4298ae9f4c894 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-6.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-6.ec2.internal,},FirstTimestamp:2026-11-02 07:23:48.207659156 +0000 UTC m=+0.360292426,LastTimestamp:2026-11-02 07:23:48.207659156 +0000 UTC m=+0.360292426,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-6.ec2.internal,}" Nov 02 07:23:48.251866 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.251847 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 07:23:48.251933 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.251883 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gjt6m" Nov 02 07:23:48.263644 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.263622 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gjt6m" Nov 02 07:23:48.265752 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.265684 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-6.ec2.internal.18e4298aeb9ed97f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-6.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-6.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-6.ec2.internal,},FirstTimestamp:2026-11-02 07:23:48.235581823 +0000 UTC m=+0.388215093,LastTimestamp:2026-11-02 07:23:48.235581823 +0000 UTC m=+0.388215093,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-6.ec2.internal,}" Nov 02 07:23:48.266173 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.266140 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 07:23:48.266237 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.266142 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 07:23:48.278927 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.278907 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 07:23:48.321327 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.315376 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:23:48.380908 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.380879 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 07:23:48.382080 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.382058 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 07:23:48.382080 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.382080 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 07:23:48.382245 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.382099 2577 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 07:23:48.382245 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.382139 2577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 07:23:48.398261 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.398243 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 07:23:48.416363 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.416346 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:23:48.420693 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.420663 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 07:23:48.422922 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.422910 2577 manager.go:341] "Starting Device Plugin manager" Nov 02 07:23:48.423498 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.423482 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 07:23:48.423555 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.423500 2577 server.go:87] "Starting device plugin registration server" Nov 02 07:23:48.423765 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.423755 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 07:23:48.423809 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.423766 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 07:23:48.423929 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.423849 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 07:23:48.423982 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.423936 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 07:23:48.423982 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.423945 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 07:23:48.424460 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.424442 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 07:23:48.424535 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.424499 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:23:48.475577 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.475554 2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-6.ec2.internal\" not found" node="ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.482691 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.482662 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal"] Nov 02 07:23:48.482747 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.482728 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 07:23:48.483729 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.483716 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientMemory" Nov 02 07:23:48.483811 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.483741 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 07:23:48.483811 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.483750 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientPID" Nov 02 07:23:48.485798 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.485786 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 07:23:48.485966 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.485952 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.486006 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.485986 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 07:23:48.486748 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.486732 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientMemory" Nov 02 07:23:48.486811 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.486758 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 07:23:48.486811 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.486767 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientPID" Nov 02 07:23:48.486876 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.486737 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientMemory" Nov 02 07:23:48.486876 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.486833 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 07:23:48.486876 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.486847 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientPID" Nov 02 07:23:48.488795 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.488782 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.488860 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.488804 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 07:23:48.489741 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.489723 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientMemory" Nov 02 07:23:48.489843 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.489750 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 07:23:48.489843 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.489760 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientPID" Nov 02 07:23:48.518262 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.518239 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb0aa2bb77fd4bd0612d1dffa702ab46-config\") pod \"kube-apiserver-proxy-ip-10-0-135-6.ec2.internal\" (UID: \"cb0aa2bb77fd4bd0612d1dffa702ab46\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.518350 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.518263 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c1a63f4a9c3f1c171863c543583e5bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" (UID: \"0c1a63f4a9c3f1c171863c543583e5bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.518350 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.518282 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1a63f4a9c3f1c171863c543583e5bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" (UID: \"0c1a63f4a9c3f1c171863c543583e5bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.524857 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.524839 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 07:23:48.525678 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.525659 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientMemory" Nov 02 07:23:48.525862 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.525707 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 07:23:48.525862 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.525738 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientPID" Nov 02 07:23:48.525862 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.525782 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.526533 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.526512 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-6.ec2.internal\" not found" node="ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.529314 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.529297 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-6.ec2.internal\" not found" node="ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.547423 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.547404 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.547471 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.547425 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-6.ec2.internal\": node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:23:48.585959 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.585937 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:23:48.618974 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.618951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c1a63f4a9c3f1c171863c543583e5bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" (UID: \"0c1a63f4a9c3f1c171863c543583e5bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.619058 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.618979 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1a63f4a9c3f1c171863c543583e5bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" (UID: \"0c1a63f4a9c3f1c171863c543583e5bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.619058 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.618995 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb0aa2bb77fd4bd0612d1dffa702ab46-config\") pod \"kube-apiserver-proxy-ip-10-0-135-6.ec2.internal\" (UID: \"cb0aa2bb77fd4bd0612d1dffa702ab46\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.619058 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.619029 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb0aa2bb77fd4bd0612d1dffa702ab46-config\") pod \"kube-apiserver-proxy-ip-10-0-135-6.ec2.internal\" (UID: \"cb0aa2bb77fd4bd0612d1dffa702ab46\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.619058 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.619040 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c1a63f4a9c3f1c171863c543583e5bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" (UID: \"0c1a63f4a9c3f1c171863c543583e5bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.619058 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.619051 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1a63f4a9c3f1c171863c543583e5bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" (UID: \"0c1a63f4a9c3f1c171863c543583e5bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.687080 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.687012 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:23:48.787878 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.787851 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:23:48.829491 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.829470 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.832586 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:48.832568 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" Nov 02 07:23:48.888464 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.888436 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:23:48.989001 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:48.988919 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:23:49.089365 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:49.089183 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:23:49.151394 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.151359 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 07:23:49.151840 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.151492 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 07:23:49.189793 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:49.189763 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:23:49.268620 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.268534 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 07:18:48 +0000 UTC" deadline="2028-07-27 23:38:06.799315112 +0000 UTC" Nov 02 07:23:49.268620 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.268572 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15208h14m17.53074693s" Nov 02 07:23:49.290026 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:49.289994 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:23:49.299119 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.299090 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 07:23:49.315686 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.315666 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:23:49.340368 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.340223 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 07:23:49.346772 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.346745 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" Nov 02 07:23:49.356335 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:49.356309 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0aa2bb77fd4bd0612d1dffa702ab46.slice/crio-82938c87d260f19af2189bdcbedcd4f36a06bb6d0d4719ea1975bc288ba4ae72 WatchSource:0}: Error finding container 82938c87d260f19af2189bdcbedcd4f36a06bb6d0d4719ea1975bc288ba4ae72: Status 404 returned error can't find the container with id 82938c87d260f19af2189bdcbedcd4f36a06bb6d0d4719ea1975bc288ba4ae72 Nov 02 07:23:49.361158 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.361142 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 07:23:49.365726 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:49.365702 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1a63f4a9c3f1c171863c543583e5bd.slice/crio-652ddf883f834360985f8af6947f12b2139cdbdff0c5cf725fcd714a869d2d2a WatchSource:0}: Error finding container 652ddf883f834360985f8af6947f12b2139cdbdff0c5cf725fcd714a869d2d2a: Status 404 returned error can't find the container with id 652ddf883f834360985f8af6947f12b2139cdbdff0c5cf725fcd714a869d2d2a Nov 02 07:23:49.371523 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.371505 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 07:23:49.385442 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.385397 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" event={"ID":"cb0aa2bb77fd4bd0612d1dffa702ab46","Type":"ContainerStarted","Data":"82938c87d260f19af2189bdcbedcd4f36a06bb6d0d4719ea1975bc288ba4ae72"} Nov 02 07:23:49.386482 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.386459 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" event={"ID":"0c1a63f4a9c3f1c171863c543583e5bd","Type":"ContainerStarted","Data":"652ddf883f834360985f8af6947f12b2139cdbdff0c5cf725fcd714a869d2d2a"} Nov 02 07:23:49.757751 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.757672 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 07:23:49.836399 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:49.836366 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 07:23:50.097013 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.096933 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 07:23:50.192851 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.192813 2577 apiserver.go:52] "Watching apiserver" Nov 02 07:23:50.216141 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.216111 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 07:23:50.222650 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.222622 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 07:23:50.223018 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.222986 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-qkcvh","openshift-multus/network-metrics-daemon-v6hrz","kube-system/global-pull-secret-syncer-9ztwm","kube-system/konnectivity-agent-5g4ck","openshift-dns/node-resolver-f6f9k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal","openshift-network-diagnostics/network-check-target-25f6d","openshift-image-registry/node-ca-72ngx","openshift-network-operator/iptables-alerter-4hq85","openshift-ovn-kubernetes/ovnkube-node-j948j","kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4","openshift-multus/multus-pm58s","openshift-cluster-node-tuning-operator/tuned-ts4xm"] Nov 02 07:23:50.225700 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.225668 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:23:50.225792 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.225752 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:23:50.227995 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.227976 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.230283 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.230183 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:50.230283 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.230253 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:23:50.232409 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.232390 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:23:50.234688 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.234671 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:23:50.236930 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.236905 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.239222 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.239198 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:23:50.242128 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.242106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.245028 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.244989 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 07:23:50.245127 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.245042 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 07:23:50.245127 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.245075 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 07:23:50.245277 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.245196 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 07:23:50.245336 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.245310 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7q84k\"" Nov 02 07:23:50.245336 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.245323 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:50.245434 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.245388 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:23:50.247904 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.247859 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7nffm\"" Nov 02 07:23:50.248075 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.248059 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 07:23:50.248180 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.248146 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 07:23:50.248292 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.248270 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.251828 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.251804 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 07:23:50.251934 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.251839 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfpr4\"" Nov 02 07:23:50.251934 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.251853 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 07:23:50.253207 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.253190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:23:50.254181 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.254147 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 07:23:50.254272 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.254231 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 07:23:50.254370 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.254351 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w2bbn\"" Nov 02 07:23:50.255274 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.255256 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 07:23:50.255775 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.255756 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.268031 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.268008 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 07:23:50.268151 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.268088 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xzmb9" Nov 02 07:23:50.280276 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.280233 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xzmb9" Nov 02 07:23:50.316554 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.316530 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 07:23:50.330400 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-conf-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.330400 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-kubernetes\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.330400 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330305 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-node-log\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.330400 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-ovnkube-config\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.330400 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-device-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.330400 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330413 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2fk\" (UniqueName: \"kubernetes.io/projected/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-kube-api-access-cg2fk\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:23:50.330819 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330487 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-cnibin\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.330819 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.330819 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330552 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj2mr\" (UniqueName: \"kubernetes.io/projected/71998a34-5575-4ae0-8892-2c9e88b1b396-kube-api-access-pj2mr\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:50.330819 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330569 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-run-ovn-kubernetes\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.330819 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330609 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-iptables-alerter-script\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:23:50.330819 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xftv\" (UniqueName: \"kubernetes.io/projected/e229a07c-7415-4af9-a9eb-9e7984e0a510-kube-api-access-2xftv\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:23:50.330819 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-env-overrides\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.330819 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-netns\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.330819 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330741 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysconfig\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.330819 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330774 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-cni-bin\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.330819 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330812 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-registration-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.331285 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330844 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-cni-multus\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.331285 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-system-cni-dir\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.331285 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330942 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-sys-fs\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.331285 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.330997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-cnibin\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.331285 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331015 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-daemon-config\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.331285 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-systemd\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.331285 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331121 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.331285 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331150 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-kubelet\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.331285 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysctl-conf\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.331285 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7m65\" (UniqueName: \"kubernetes.io/projected/3201efd6-0d33-4323-bab0-b76338d090ed-kube-api-access-c7m65\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.331285 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-hostroot\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.331285 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331257 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e229a07c-7415-4af9-a9eb-9e7984e0a510-host\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:23:50.331285 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-var-lib-kubelet\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3201efd6-0d33-4323-bab0-b76338d090ed-etc-tuned\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-cni-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e229a07c-7415-4af9-a9eb-9e7984e0a510-serviceca\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6e945a4-9c6d-4310-902f-eff8fc826621-agent-certs\") pod \"konnectivity-agent-5g4ck\" (UID: \"d6e945a4-9c6d-4310-902f-eff8fc826621\") " pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-hosts-file\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331522 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-run-netns\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-etc-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331593 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-cni-bin\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331628 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvtjs\" (UniqueName: \"kubernetes.io/projected/85e9ffec-236a-4a8c-8dd1-7014e032e180-kube-api-access-lvtjs\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6e945a4-9c6d-4310-902f-eff8fc826621-konnectivity-ca\") pod \"konnectivity-agent-5g4ck\" (UID: \"d6e945a4-9c6d-4310-902f-eff8fc826621\") " pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-sys\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331775 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/973a9b8f-405e-4dc4-8322-4d12a152178d-ovn-node-metrics-cert\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331801 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/253a4509-48e5-48c2-8c13-2dc361e5763f-dbus\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:50.331875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-tmp-dir\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-lib-modules\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.331946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3201efd6-0d33-4323-bab0-b76338d090ed-tmp\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-slash\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-cni-netd\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-socket-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85e9ffec-236a-4a8c-8dd1-7014e032e180-cni-binary-copy\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332121 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332147 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-host-slash\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332174 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djnzm\" (UniqueName: \"kubernetes.io/projected/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-kube-api-access-djnzm\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-modprobe-d\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-host\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332390 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-socket-dir-parent\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332414 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-k8s-cni-cncf-io\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.332589 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysctl-d\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-log-socket\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-ovnkube-script-lib\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6lln\" (UniqueName: \"kubernetes.io/projected/973a9b8f-405e-4dc4-8322-4d12a152178d-kube-api-access-p6lln\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wkph\" (UniqueName: \"kubernetes.io/projected/a1349fb3-19b0-4648-92e1-fcd2da2076bf-kube-api-access-5wkph\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-multus-certs\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332645 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-systemd-units\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332670 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-var-lib-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-etc-kubernetes\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/253a4509-48e5-48c2-8c13-2dc361e5763f-kubelet-config\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332859 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-659fv\" (UniqueName: \"kubernetes.io/projected/2eeba619-ec6d-4ec8-828b-2f409d53630d-kube-api-access-659fv\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-systemd\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8qfz\" (UniqueName: \"kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz\") pod \"network-check-target-25f6d\" (UID: \"ef891d1c-0c5c-45b3-98eb-563324d8822f\") " pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-kubelet\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.333281 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.332974 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.333867 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.333001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-ovn\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.333867 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.333026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-run\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.333867 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.333055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-system-cni-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.354530 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.354380 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 07:23:50.354530 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.354453 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 07:23:50.354781 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.354763 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 07:23:50.354836 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.354809 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 07:23:50.354914 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.354898 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355146 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355193 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c9gk6\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355208 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355236 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355243 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-szw6v\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355275 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355286 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lptml\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355299 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355276 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355212 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355212 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355329 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tc9dv\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355325 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355320 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 07:23:50.355374 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355381 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6xxpk\"" Nov 02 07:23:50.356061 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.355428 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 07:23:50.433494 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qfz\" (UniqueName: \"kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz\") pod \"network-check-target-25f6d\" (UID: \"ef891d1c-0c5c-45b3-98eb-563324d8822f\") " pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:23:50.433494 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-kubelet\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.433740 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.433740 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-ovn\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.433740 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-run\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.433740 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-system-cni-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.433740 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433616 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-conf-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.433740 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-kubernetes\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.433740 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-node-log\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.433740 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-ovnkube-config\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.433740 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-device-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2fk\" (UniqueName: \"kubernetes.io/projected/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-kube-api-access-cg2fk\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-cnibin\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433796 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj2mr\" (UniqueName: \"kubernetes.io/projected/71998a34-5575-4ae0-8892-2c9e88b1b396-kube-api-access-pj2mr\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-run-ovn-kubernetes\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433871 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-iptables-alerter-script\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xftv\" (UniqueName: \"kubernetes.io/projected/e229a07c-7415-4af9-a9eb-9e7984e0a510-kube-api-access-2xftv\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433925 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-env-overrides\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433953 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-netns\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.433957 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-kubernetes\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434009 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-netns\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434043 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-node-log\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434084 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-kubelet\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434123 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.434205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434164 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-ovn\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434302 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-run\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434362 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-system-cni-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434405 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-conf-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysconfig\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434460 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-cni-bin\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-registration-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-cni-multus\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-system-cni-dir\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-sys-fs\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-cnibin\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434586 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-daemon-config\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-systemd\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434618 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-kubelet\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysctl-conf\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7m65\" (UniqueName: \"kubernetes.io/projected/3201efd6-0d33-4323-bab0-b76338d090ed-kube-api-access-c7m65\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.434949 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434743 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-device-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-hostroot\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e229a07c-7415-4af9-a9eb-9e7984e0a510-host\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-var-lib-kubelet\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434843 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3201efd6-0d33-4323-bab0-b76338d090ed-etc-tuned\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-cni-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e229a07c-7415-4af9-a9eb-9e7984e0a510-serviceca\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6e945a4-9c6d-4310-902f-eff8fc826621-agent-certs\") pod \"konnectivity-agent-5g4ck\" (UID: \"d6e945a4-9c6d-4310-902f-eff8fc826621\") " pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.434982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-hosts-file\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-run-netns\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435015 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-cnibin\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-etc-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-cni-bin\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvtjs\" (UniqueName: \"kubernetes.io/projected/85e9ffec-236a-4a8c-8dd1-7014e032e180-kube-api-access-lvtjs\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435247 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-ovnkube-config\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435293 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysctl-conf\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435361 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-kubelet\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.436621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6e945a4-9c6d-4310-902f-eff8fc826621-konnectivity-ca\") pod \"konnectivity-agent-5g4ck\" (UID: \"d6e945a4-9c6d-4310-902f-eff8fc826621\") " pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435396 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-sys\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435408 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-systemd\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435415 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/973a9b8f-405e-4dc4-8322-4d12a152178d-ovn-node-metrics-cert\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435431 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-hostroot\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435468 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-registration-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435498 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-system-cni-dir\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435521 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e229a07c-7415-4af9-a9eb-9e7984e0a510-host\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435538 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysconfig\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435579 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-sys-fs\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435557 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435648 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-daemon-config\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435650 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435660 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-cni-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435688 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-cni-bin\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435731 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-cnibin\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435748 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-var-lib-kubelet\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.437565 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435790 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-hosts-file\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:23:50.438461 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.435825 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-run-netns\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.438461 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.437896 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.440722 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.440337 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-run-ovn-kubernetes\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.440722 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.440553 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-etc-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.440722 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.440648 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-sys\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.440722 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.440686 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.440964 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.440707 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/973a9b8f-405e-4dc4-8322-4d12a152178d-ovn-node-metrics-cert\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.440964 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.440757 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-cni-multus\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.440964 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.440902 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6e945a4-9c6d-4310-902f-eff8fc826621-agent-certs\") pod \"konnectivity-agent-5g4ck\" (UID: \"d6e945a4-9c6d-4310-902f-eff8fc826621\") " pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:23:50.441107 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441055 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.441107 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441068 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-env-overrides\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.441265 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441237 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/253a4509-48e5-48c2-8c13-2dc361e5763f-dbus\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:50.441383 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-tmp-dir\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:23:50.441383 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441308 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e229a07c-7415-4af9-a9eb-9e7984e0a510-serviceca\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:23:50.441383 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441325 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6e945a4-9c6d-4310-902f-eff8fc826621-konnectivity-ca\") pod \"konnectivity-agent-5g4ck\" (UID: \"d6e945a4-9c6d-4310-902f-eff8fc826621\") " pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:23:50.441383 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-lib-modules\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.441383 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3201efd6-0d33-4323-bab0-b76338d090ed-tmp\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.441668 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441405 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/253a4509-48e5-48c2-8c13-2dc361e5763f-dbus\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:50.441668 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-slash\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.441668 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-cni-netd\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.441668 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-socket-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.441668 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85e9ffec-236a-4a8c-8dd1-7014e032e180-cni-binary-copy\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.441668 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441554 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-lib-modules\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.441668 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:50.441668 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441612 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-host-slash\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:23:50.441668 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.441654 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 07:23:50.441668 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.441668 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441665 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-tmp-dir\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnzm\" (UniqueName: \"kubernetes.io/projected/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-kube-api-access-djnzm\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-modprobe-d\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441748 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-cni-netd\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-host\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-socket-dir-parent\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-k8s-cni-cncf-io\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.441907 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-cni-bin\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.441993 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret podName:253a4509-48e5-48c2-8c13-2dc361e5763f nodeName:}" failed. No retries permitted until 2026-11-02 07:23:50.941967033 +0000 UTC m=+3.094600291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret") pod "global-pull-secret-syncer-9ztwm" (UID: "253a4509-48e5-48c2-8c13-2dc361e5763f") : object "kube-system"/"original-pull-secret" not registered Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.442269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.442371 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-k8s-cni-cncf-io\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.442379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysctl-d\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.442421 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-host-slash\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.442501 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.442526 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-iptables-alerter-script\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:23:50.445782 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.442575 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs podName:71998a34-5575-4ae0-8892-2c9e88b1b396 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:50.942541412 +0000 UTC m=+3.095174684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs") pod "network-metrics-daemon-v6hrz" (UID: "71998a34-5575-4ae0-8892-2c9e88b1b396") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.442625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-log-socket\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.442683 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-ovnkube-script-lib\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.442713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6lln\" (UniqueName: \"kubernetes.io/projected/973a9b8f-405e-4dc4-8322-4d12a152178d-kube-api-access-p6lln\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.442749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wkph\" (UniqueName: \"kubernetes.io/projected/a1349fb3-19b0-4648-92e1-fcd2da2076bf-kube-api-access-5wkph\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.442848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-multus-certs\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.442904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-systemd-units\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.442940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-var-lib-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.442975 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443037 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-etc-kubernetes\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/253a4509-48e5-48c2-8c13-2dc361e5763f-kubelet-config\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-659fv\" (UniqueName: \"kubernetes.io/projected/2eeba619-ec6d-4ec8-828b-2f409d53630d-kube-api-access-659fv\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-systemd\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443409 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443478 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-host\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443513 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-systemd\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443556 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-modprobe-d\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.446663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443603 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-slash\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443582 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-multus-certs\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443660 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-socket-dir-parent\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443686 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-systemd-units\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443739 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-var-lib-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443771 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-socket-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443812 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-ovnkube-script-lib\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.443825 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.444441 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-etc-kubernetes\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.444671 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3201efd6-0d33-4323-bab0-b76338d090ed-etc-tuned\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.445613 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.446396 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/253a4509-48e5-48c2-8c13-2dc361e5763f-kubelet-config\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.446446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85e9ffec-236a-4a8c-8dd1-7014e032e180-cni-binary-copy\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.446523 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-log-socket\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.447558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.446540 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysctl-d\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.448287 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.447683 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3201efd6-0d33-4323-bab0-b76338d090ed-tmp\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.455782 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.455761 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 07:23:50.455782 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.455784 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 07:23:50.455963 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.455799 2577 projected.go:196] Error preparing data for projected volume kube-api-access-w8qfz for pod openshift-network-diagnostics/network-check-target-25f6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:23:50.455963 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.455855 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz podName:ef891d1c-0c5c-45b3-98eb-563324d8822f nodeName:}" failed. No retries permitted until 2026-11-02 07:23:50.955837379 +0000 UTC m=+3.108470638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w8qfz" (UniqueName: "kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz") pod "network-check-target-25f6d" (UID: "ef891d1c-0c5c-45b3-98eb-563324d8822f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:23:50.459430 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.459407 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7m65\" (UniqueName: \"kubernetes.io/projected/3201efd6-0d33-4323-bab0-b76338d090ed-kube-api-access-c7m65\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.460007 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.459980 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2fk\" (UniqueName: \"kubernetes.io/projected/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-kube-api-access-cg2fk\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:23:50.461364 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.461338 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xftv\" (UniqueName: \"kubernetes.io/projected/e229a07c-7415-4af9-a9eb-9e7984e0a510-kube-api-access-2xftv\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:23:50.464475 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.464455 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvtjs\" (UniqueName: \"kubernetes.io/projected/85e9ffec-236a-4a8c-8dd1-7014e032e180-kube-api-access-lvtjs\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.470420 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.470305 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-659fv\" (UniqueName: \"kubernetes.io/projected/2eeba619-ec6d-4ec8-828b-2f409d53630d-kube-api-access-659fv\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.470616 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.470585 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wkph\" (UniqueName: \"kubernetes.io/projected/a1349fb3-19b0-4648-92e1-fcd2da2076bf-kube-api-access-5wkph\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.472076 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.472048 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6lln\" (UniqueName: \"kubernetes.io/projected/973a9b8f-405e-4dc4-8322-4d12a152178d-kube-api-access-p6lln\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.472220 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.472184 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj2mr\" (UniqueName: \"kubernetes.io/projected/71998a34-5575-4ae0-8892-2c9e88b1b396-kube-api-access-pj2mr\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:50.473493 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.473474 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnzm\" (UniqueName: \"kubernetes.io/projected/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-kube-api-access-djnzm\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:23:50.541656 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.541622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pm58s" Nov 02 07:23:50.547468 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.547441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:23:50.557962 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:50.557934 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e9ffec_236a_4a8c_8dd1_7014e032e180.slice/crio-78d4343c996b21ad40631bc0d272b81a513f5afc39e60b05292139db33c19d15 WatchSource:0}: Error finding container 78d4343c996b21ad40631bc0d272b81a513f5afc39e60b05292139db33c19d15: Status 404 returned error can't find the container with id 78d4343c996b21ad40631bc0d272b81a513f5afc39e60b05292139db33c19d15 Nov 02 07:23:50.558841 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:50.558814 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e945a4_9c6d_4310_902f_eff8fc826621.slice/crio-3a2d1c0c22f3fd36ac5cc56b309fac4676b439e476c00aa38ae5c6236cc89eb1 WatchSource:0}: Error finding container 3a2d1c0c22f3fd36ac5cc56b309fac4676b439e476c00aa38ae5c6236cc89eb1: Status 404 returned error can't find the container with id 3a2d1c0c22f3fd36ac5cc56b309fac4676b439e476c00aa38ae5c6236cc89eb1 Nov 02 07:23:50.560691 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.560670 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:23:50.565790 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.565766 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:23:50.572355 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.572334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:23:50.573633 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:50.573596 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77af1f03_6bc5_4dad_8646_eae4f5ae40f0.slice/crio-c5b94dbfaf37d5068efe781f58c18c6f943469de2799ebe465ad7e2411c93af4 WatchSource:0}: Error finding container c5b94dbfaf37d5068efe781f58c18c6f943469de2799ebe465ad7e2411c93af4: Status 404 returned error can't find the container with id c5b94dbfaf37d5068efe781f58c18c6f943469de2799ebe465ad7e2411c93af4 Nov 02 07:23:50.577022 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:50.576993 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb3f2d5_5e21_48c3_af74_eb8fd530046a.slice/crio-545680a331c78bec9728827fd32a150b095c3cc57a23b2dc24cc15a2ec90a9be WatchSource:0}: Error finding container 545680a331c78bec9728827fd32a150b095c3cc57a23b2dc24cc15a2ec90a9be: Status 404 returned error can't find the container with id 545680a331c78bec9728827fd32a150b095c3cc57a23b2dc24cc15a2ec90a9be Nov 02 07:23:50.582706 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:50.582682 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3201efd6_0d33_4323_bab0_b76338d090ed.slice/crio-1d8261f550c8a5f5462570a037d715211d0064c8466f519f8cec1dea0849a647 WatchSource:0}: Error finding container 1d8261f550c8a5f5462570a037d715211d0064c8466f519f8cec1dea0849a647: Status 404 returned error can't find the container with id 1d8261f550c8a5f5462570a037d715211d0064c8466f519f8cec1dea0849a647 Nov 02 07:23:50.596997 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.596804 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:23:50.598964 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.598920 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:23:50.609343 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:50.609222 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eeba619_ec6d_4ec8_828b_2f409d53630d.slice/crio-9cb3cd7090bb43620924b05554a98a225161bf986b525b7c7da5992d7b25a593 WatchSource:0}: Error finding container 9cb3cd7090bb43620924b05554a98a225161bf986b525b7c7da5992d7b25a593: Status 404 returned error can't find the container with id 9cb3cd7090bb43620924b05554a98a225161bf986b525b7c7da5992d7b25a593 Nov 02 07:23:50.611090 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:50.611039 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode229a07c_7415_4af9_a9eb_9e7984e0a510.slice/crio-2a2d3fbe31932c9965beea105f1435af6f4ccf22e75ffd6c5199e42e257851ef WatchSource:0}: Error finding container 2a2d3fbe31932c9965beea105f1435af6f4ccf22e75ffd6c5199e42e257851ef: Status 404 returned error can't find the container with id 2a2d3fbe31932c9965beea105f1435af6f4ccf22e75ffd6c5199e42e257851ef Nov 02 07:23:50.613537 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.613514 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:23:50.620311 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.620287 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:23:50.624837 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:50.624811 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1349fb3_19b0_4648_92e1_fcd2da2076bf.slice/crio-2e47d4a08d3395d94cd3cc3f03529bb79e42c8df4074f3ea97ae2f4c5f130336 WatchSource:0}: Error finding container 2e47d4a08d3395d94cd3cc3f03529bb79e42c8df4074f3ea97ae2f4c5f130336: Status 404 returned error can't find the container with id 2e47d4a08d3395d94cd3cc3f03529bb79e42c8df4074f3ea97ae2f4c5f130336 Nov 02 07:23:50.630891 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:23:50.630858 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod973a9b8f_405e_4dc4_8322_4d12a152178d.slice/crio-7dbf748226a4d9ef4d3fcb6b13a8eb00667daf6284bc74941652929074bc9dcb WatchSource:0}: Error finding container 7dbf748226a4d9ef4d3fcb6b13a8eb00667daf6284bc74941652929074bc9dcb: Status 404 returned error can't find the container with id 7dbf748226a4d9ef4d3fcb6b13a8eb00667daf6284bc74941652929074bc9dcb Nov 02 07:23:50.947941 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.947916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:50.948040 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:50.947981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:50.948081 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.948068 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 07:23:50.948119 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.948077 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:23:50.948174 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.948135 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret podName:253a4509-48e5-48c2-8c13-2dc361e5763f nodeName:}" failed. No retries permitted until 2026-11-02 07:23:51.948116227 +0000 UTC m=+4.100749486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret") pod "global-pull-secret-syncer-9ztwm" (UID: "253a4509-48e5-48c2-8c13-2dc361e5763f") : object "kube-system"/"original-pull-secret" not registered Nov 02 07:23:50.948174 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:50.948158 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs podName:71998a34-5575-4ae0-8892-2c9e88b1b396 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:51.948145615 +0000 UTC m=+4.100778873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs") pod "network-metrics-daemon-v6hrz" (UID: "71998a34-5575-4ae0-8892-2c9e88b1b396") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:23:51.048534 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.048502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qfz\" (UniqueName: \"kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz\") pod \"network-check-target-25f6d\" (UID: \"ef891d1c-0c5c-45b3-98eb-563324d8822f\") " pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:23:51.048680 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:51.048665 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 07:23:51.048752 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:51.048689 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 07:23:51.048752 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:51.048702 2577 projected.go:196] Error preparing data for projected volume kube-api-access-w8qfz for pod openshift-network-diagnostics/network-check-target-25f6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:23:51.048864 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:51.048757 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz podName:ef891d1c-0c5c-45b3-98eb-563324d8822f nodeName:}" failed. No retries permitted until 2026-11-02 07:23:52.048739213 +0000 UTC m=+4.201372472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w8qfz" (UniqueName: "kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz") pod "network-check-target-25f6d" (UID: "ef891d1c-0c5c-45b3-98eb-563324d8822f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:23:51.284673 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.284558 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 07:18:50 +0000 UTC" deadline="2028-06-04 23:24:13.644276994 +0000 UTC" Nov 02 07:23:51.284673 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.284625 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13936h0m22.359656215s" Nov 02 07:23:51.419052 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.418929 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" event={"ID":"3201efd6-0d33-4323-bab0-b76338d090ed","Type":"ContainerStarted","Data":"1d8261f550c8a5f5462570a037d715211d0064c8466f519f8cec1dea0849a647"} Nov 02 07:23:51.426123 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.425953 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4hq85" event={"ID":"77af1f03-6bc5-4dad-8646-eae4f5ae40f0","Type":"ContainerStarted","Data":"c5b94dbfaf37d5068efe781f58c18c6f943469de2799ebe465ad7e2411c93af4"} Nov 02 07:23:51.431055 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.431023 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f6f9k" event={"ID":"fdb3f2d5-5e21-48c3-af74-eb8fd530046a","Type":"ContainerStarted","Data":"545680a331c78bec9728827fd32a150b095c3cc57a23b2dc24cc15a2ec90a9be"} Nov 02 07:23:51.446627 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.446594 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5g4ck" event={"ID":"d6e945a4-9c6d-4310-902f-eff8fc826621","Type":"ContainerStarted","Data":"3a2d1c0c22f3fd36ac5cc56b309fac4676b439e476c00aa38ae5c6236cc89eb1"} Nov 02 07:23:51.463703 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.463671 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pm58s" event={"ID":"85e9ffec-236a-4a8c-8dd1-7014e032e180","Type":"ContainerStarted","Data":"78d4343c996b21ad40631bc0d272b81a513f5afc39e60b05292139db33c19d15"} Nov 02 07:23:51.479323 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.478421 2577 generic.go:356] "Generic (PLEG): container finished" podID="0c1a63f4a9c3f1c171863c543583e5bd" containerID="deda5e4a079b04917daf0b9178ea884cde825ec062a5af1f91efa2266ea9ba7e" exitCode=0 Nov 02 07:23:51.479323 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.478512 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" event={"ID":"0c1a63f4a9c3f1c171863c543583e5bd","Type":"ContainerDied","Data":"deda5e4a079b04917daf0b9178ea884cde825ec062a5af1f91efa2266ea9ba7e"} Nov 02 07:23:51.487666 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.487275 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" event={"ID":"cb0aa2bb77fd4bd0612d1dffa702ab46","Type":"ContainerStarted","Data":"a35e2331b24fadd9bdf4ffd0375d0689c18b7b57ce1f237c690c110bfb6eee39"} Nov 02 07:23:51.491780 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.491754 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" event={"ID":"973a9b8f-405e-4dc4-8322-4d12a152178d","Type":"ContainerStarted","Data":"7dbf748226a4d9ef4d3fcb6b13a8eb00667daf6284bc74941652929074bc9dcb"} Nov 02 07:23:51.502494 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.502268 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" event={"ID":"a1349fb3-19b0-4648-92e1-fcd2da2076bf","Type":"ContainerStarted","Data":"2e47d4a08d3395d94cd3cc3f03529bb79e42c8df4074f3ea97ae2f4c5f130336"} Nov 02 07:23:51.514496 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.514466 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-72ngx" event={"ID":"e229a07c-7415-4af9-a9eb-9e7984e0a510","Type":"ContainerStarted","Data":"2a2d3fbe31932c9965beea105f1435af6f4ccf22e75ffd6c5199e42e257851ef"} Nov 02 07:23:51.518148 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.518118 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkcvh" event={"ID":"2eeba619-ec6d-4ec8-828b-2f409d53630d","Type":"ContainerStarted","Data":"9cb3cd7090bb43620924b05554a98a225161bf986b525b7c7da5992d7b25a593"} Nov 02 07:23:51.558959 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.558857 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" podStartSLOduration=2.55884466 podStartE2EDuration="2.55884466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:51.55850411 +0000 UTC m=+3.711137390" watchObservedRunningTime="2026-11-02 07:23:51.55884466 +0000 UTC m=+3.711477939" Nov 02 07:23:51.955900 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.955810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:51.955900 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:51.955871 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:51.956137 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:51.956012 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:23:51.956137 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:51.956078 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs podName:71998a34-5575-4ae0-8892-2c9e88b1b396 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:53.956058968 +0000 UTC m=+6.108692243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs") pod "network-metrics-daemon-v6hrz" (UID: "71998a34-5575-4ae0-8892-2c9e88b1b396") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:23:51.956328 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:51.956147 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 07:23:51.956328 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:51.956197 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret podName:253a4509-48e5-48c2-8c13-2dc361e5763f nodeName:}" failed. No retries permitted until 2026-11-02 07:23:53.956171574 +0000 UTC m=+6.108804831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret") pod "global-pull-secret-syncer-9ztwm" (UID: "253a4509-48e5-48c2-8c13-2dc361e5763f") : object "kube-system"/"original-pull-secret" not registered Nov 02 07:23:52.056641 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:52.056592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qfz\" (UniqueName: \"kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz\") pod \"network-check-target-25f6d\" (UID: \"ef891d1c-0c5c-45b3-98eb-563324d8822f\") " pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:23:52.056878 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:52.056785 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 07:23:52.056878 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:52.056805 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 07:23:52.056878 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:52.056819 2577 projected.go:196] Error preparing data for projected volume kube-api-access-w8qfz for pod openshift-network-diagnostics/network-check-target-25f6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:23:52.057044 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:52.056884 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz podName:ef891d1c-0c5c-45b3-98eb-563324d8822f nodeName:}" failed. No retries permitted until 2026-11-02 07:23:54.056857136 +0000 UTC m=+6.209490397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w8qfz" (UniqueName: "kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz") pod "network-check-target-25f6d" (UID: "ef891d1c-0c5c-45b3-98eb-563324d8822f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:23:52.285250 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:52.285120 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 07:18:50 +0000 UTC" deadline="2028-06-01 08:07:01.473671858 +0000 UTC" Nov 02 07:23:52.285250 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:52.285173 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13848h43m9.188516024s" Nov 02 07:23:52.384910 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:52.384419 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:23:52.384910 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:52.384442 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:52.384910 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:52.384554 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:23:52.384910 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:52.384613 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:23:52.384910 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:52.384674 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:52.384910 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:52.384748 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:23:52.524287 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:52.524249 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" event={"ID":"0c1a63f4a9c3f1c171863c543583e5bd","Type":"ContainerStarted","Data":"5c77504387a2204768dbe9024e5a32f0eccba8fc3122097ec4612830c60aadc5"} Nov 02 07:23:52.568448 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:52.567282 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" podStartSLOduration=3.567261856 podStartE2EDuration="3.567261856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:52.56629792 +0000 UTC m=+4.718931199" watchObservedRunningTime="2026-11-02 07:23:52.567261856 +0000 UTC m=+4.719895135" Nov 02 07:23:53.567860 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:53.567509 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkcvh" event={"ID":"2eeba619-ec6d-4ec8-828b-2f409d53630d","Type":"ContainerStarted","Data":"51fe96cb2c356fff6a19f73d3226016f9d910240087bcfd7bf776b33c2144b2a"} Nov 02 07:23:53.973734 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:53.973690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:53.973911 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:53.973742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:53.973911 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:53.973889 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:23:53.974025 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:53.973950 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs podName:71998a34-5575-4ae0-8892-2c9e88b1b396 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:57.973931345 +0000 UTC m=+10.126564602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs") pod "network-metrics-daemon-v6hrz" (UID: "71998a34-5575-4ae0-8892-2c9e88b1b396") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:23:53.974188 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:53.974119 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 07:23:53.974188 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:53.974184 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret podName:253a4509-48e5-48c2-8c13-2dc361e5763f nodeName:}" failed. No retries permitted until 2026-11-02 07:23:57.974155158 +0000 UTC m=+10.126788414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret") pod "global-pull-secret-syncer-9ztwm" (UID: "253a4509-48e5-48c2-8c13-2dc361e5763f") : object "kube-system"/"original-pull-secret" not registered Nov 02 07:23:54.074394 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:54.074354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qfz\" (UniqueName: \"kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz\") pod \"network-check-target-25f6d\" (UID: \"ef891d1c-0c5c-45b3-98eb-563324d8822f\") " pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:23:54.074635 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:54.074587 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 07:23:54.074635 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:54.074607 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 07:23:54.074635 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:54.074619 2577 projected.go:196] Error preparing data for projected volume kube-api-access-w8qfz for pod openshift-network-diagnostics/network-check-target-25f6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:23:54.074840 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:54.074679 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz podName:ef891d1c-0c5c-45b3-98eb-563324d8822f nodeName:}" failed. No retries permitted until 2026-11-02 07:23:58.074662406 +0000 UTC m=+10.227295669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w8qfz" (UniqueName: "kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz") pod "network-check-target-25f6d" (UID: "ef891d1c-0c5c-45b3-98eb-563324d8822f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:23:54.385707 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:54.385638 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:54.385864 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:54.385770 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:23:54.385864 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:54.385850 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:54.385979 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:54.385938 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:23:54.386026 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:54.385995 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:23:54.386076 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:54.386062 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:23:54.574006 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:54.573910 2577 generic.go:356] "Generic (PLEG): container finished" podID="2eeba619-ec6d-4ec8-828b-2f409d53630d" containerID="51fe96cb2c356fff6a19f73d3226016f9d910240087bcfd7bf776b33c2144b2a" exitCode=0 Nov 02 07:23:54.574006 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:54.574000 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkcvh" event={"ID":"2eeba619-ec6d-4ec8-828b-2f409d53630d","Type":"ContainerDied","Data":"51fe96cb2c356fff6a19f73d3226016f9d910240087bcfd7bf776b33c2144b2a"} Nov 02 07:23:54.579784 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:54.579317 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5g4ck" event={"ID":"d6e945a4-9c6d-4310-902f-eff8fc826621","Type":"ContainerStarted","Data":"560f44d4628265df42b7a4b0c7f5ac005f52bd37e4fe53f3ea33e19c94354633"} Nov 02 07:23:55.589329 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:55.589238 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-72ngx" event={"ID":"e229a07c-7415-4af9-a9eb-9e7984e0a510","Type":"ContainerStarted","Data":"02094d01c2fce18ebfc89ce974d08b6855eebc26237c1d73d50da13925823724"} Nov 02 07:23:55.627179 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:55.627114 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5g4ck" podStartSLOduration=3.875926459 podStartE2EDuration="7.627098482s" totalImagesPullingTime="3.751172023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:54.670837562 +0000 UTC m=+6.823470841" watchObservedRunningTime="2026-11-02 07:23:55.627098482 +0000 UTC m=+7.779731754" Nov 02 07:23:55.627414 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:55.627384 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-72ngx" podStartSLOduration=2.915787448 podStartE2EDuration="7.627376541s" totalImagesPullingTime="4.711589093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:55.627289884 +0000 UTC m=+7.779923163" watchObservedRunningTime="2026-11-02 07:23:55.627376541 +0000 UTC m=+7.780009819" Nov 02 07:23:56.384849 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:56.384811 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:23:56.385008 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:56.384931 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:23:56.385320 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:56.385301 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:56.385443 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:56.385403 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:23:56.385500 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:56.385457 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:56.385550 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:56.385535 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:23:56.593582 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:56.593546 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" event={"ID":"a1349fb3-19b0-4648-92e1-fcd2da2076bf","Type":"ContainerStarted","Data":"e682857363a7fdef38d610aac4516da27ff25bf8b25ec4d55118892eae04dabb"} Nov 02 07:23:57.684401 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:57.684345 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 07:23:58.006513 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.006452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:58.006513 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.006487 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:58.006676 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:58.006614 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:23:58.006676 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:58.006665 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs podName:71998a34-5575-4ae0-8892-2c9e88b1b396 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.00664701 +0000 UTC m=+18.159280266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs") pod "network-metrics-daemon-v6hrz" (UID: "71998a34-5575-4ae0-8892-2c9e88b1b396") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:23:58.006981 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:58.006966 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 07:23:58.007020 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:58.007010 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret podName:253a4509-48e5-48c2-8c13-2dc361e5763f nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.00699955 +0000 UTC m=+18.159632807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret") pod "global-pull-secret-syncer-9ztwm" (UID: "253a4509-48e5-48c2-8c13-2dc361e5763f") : object "kube-system"/"original-pull-secret" not registered Nov 02 07:23:58.107218 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.107174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qfz\" (UniqueName: \"kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz\") pod \"network-check-target-25f6d\" (UID: \"ef891d1c-0c5c-45b3-98eb-563324d8822f\") " pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:23:58.107345 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:58.107295 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 07:23:58.107345 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:58.107321 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 07:23:58.107345 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:58.107335 2577 projected.go:196] Error preparing data for projected volume kube-api-access-w8qfz for pod openshift-network-diagnostics/network-check-target-25f6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:23:58.107506 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:58.107396 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz podName:ef891d1c-0c5c-45b3-98eb-563324d8822f nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.107375702 +0000 UTC m=+18.260008974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w8qfz" (UniqueName: "kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz") pod "network-check-target-25f6d" (UID: "ef891d1c-0c5c-45b3-98eb-563324d8822f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:23:58.383489 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.383466 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:23:58.383618 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.383598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:23:58.383680 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:58.383615 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:23:58.383737 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:58.383719 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:23:58.383807 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.383771 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:23:58.383862 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:23:58.383822 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:23:58.433505 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.433374 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T07:23:57.684365127Z","UUID":"18f1b352-9ad2-4277-9240-124f1a83880e","Handler":null,"Name":"","Endpoint":""} Nov 02 07:23:58.434751 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.434733 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 07:23:58.434751 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.434756 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 07:23:58.605741 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.605710 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f6f9k" event={"ID":"fdb3f2d5-5e21-48c3-af74-eb8fd530046a","Type":"ContainerStarted","Data":"0563d428ad885a58b3f0d53934bf3400a7b68497fe227a2f2a613dcd9a2743bd"} Nov 02 07:23:58.608090 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.607998 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pm58s" event={"ID":"85e9ffec-236a-4a8c-8dd1-7014e032e180","Type":"ContainerStarted","Data":"8c7306290afe7f8a485440d358fe0d84eee66f84fabae5dd4bec66e84294cd89"} Nov 02 07:23:58.610943 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.610921 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" event={"ID":"a1349fb3-19b0-4648-92e1-fcd2da2076bf","Type":"ContainerStarted","Data":"3134d3d84524de12ac503602ec74662be6b3ab529256272b0c5daac6c73c0c0c"} Nov 02 07:23:58.717791 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.717209 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pm58s" podStartSLOduration=3.225891592 podStartE2EDuration="10.717192185s" totalImagesPullingTime="7.491300593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:58.716589336 +0000 UTC m=+10.869222625" watchObservedRunningTime="2026-11-02 07:23:58.717192185 +0000 UTC m=+10.869825461" Nov 02 07:23:58.717791 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.717611 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-f6f9k" podStartSLOduration=2.9357885059999997 podStartE2EDuration="10.717604713s" totalImagesPullingTime="7.781816207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:58.671083216 +0000 UTC m=+10.823716495" watchObservedRunningTime="2026-11-02 07:23:58.717604713 +0000 UTC m=+10.870237990" Nov 02 07:23:58.733060 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.732935 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:23:58.922876 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:58.922576 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:23:59.059451 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:59.059421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:23:59.352586 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:59.352392 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:23:59.353247 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:59.353223 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:23:59.615763 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:59.615687 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" event={"ID":"a1349fb3-19b0-4648-92e1-fcd2da2076bf","Type":"ContainerStarted","Data":"01dfce8d4adf2e6f73b1b30774ec1aa44df78368eb5539bc9dbda57a4a568b81"} Nov 02 07:23:59.618180 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:59.618050 2577 generic.go:356] "Generic (PLEG): container finished" podID="2eeba619-ec6d-4ec8-828b-2f409d53630d" containerID="bd0ea034e4ba55331c7fa4eee371b0ca47d7efe47f18a50cd0a3b0003c74fc61" exitCode=0 Nov 02 07:23:59.618180 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:59.618123 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkcvh" event={"ID":"2eeba619-ec6d-4ec8-828b-2f409d53630d","Type":"ContainerDied","Data":"bd0ea034e4ba55331c7fa4eee371b0ca47d7efe47f18a50cd0a3b0003c74fc61"} Nov 02 07:23:59.621485 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:59.621434 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4hq85" event={"ID":"77af1f03-6bc5-4dad-8646-eae4f5ae40f0","Type":"ContainerStarted","Data":"3a82f9cb09b8593ce7af0d1f46a0af6e0a8731f8b35654de7f055ecba0b71364"} Nov 02 07:23:59.658001 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:59.657945 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" podStartSLOduration=3.877757625 podStartE2EDuration="11.657927968s" totalImagesPullingTime="7.780170343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:59.656994401 +0000 UTC m=+11.809627680" watchObservedRunningTime="2026-11-02 07:23:59.657927968 +0000 UTC m=+11.810561246" Nov 02 07:23:59.694918 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:23:59.694867 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4hq85" podStartSLOduration=4.060087909 podStartE2EDuration="11.694853747s" totalImagesPullingTime="7.634765838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:59.694602134 +0000 UTC m=+11.847235414" watchObservedRunningTime="2026-11-02 07:23:59.694853747 +0000 UTC m=+11.847487026" Nov 02 07:24:00.382889 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:00.382864 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:00.383236 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:00.382968 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:00.383236 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:00.383080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:00.383236 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:00.383100 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:00.383236 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:00.383191 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:00.383390 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:00.383285 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:00.526373 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:00.526319 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:00.625769 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:00.625695 2577 generic.go:356] "Generic (PLEG): container finished" podID="2eeba619-ec6d-4ec8-828b-2f409d53630d" containerID="a834a5f9b4cc559ed1af1d146ee7e52d8421b295d02a8e656c500534f3222dbc" exitCode=0 Nov 02 07:24:00.625924 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:00.625786 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkcvh" event={"ID":"2eeba619-ec6d-4ec8-828b-2f409d53630d","Type":"ContainerDied","Data":"a834a5f9b4cc559ed1af1d146ee7e52d8421b295d02a8e656c500534f3222dbc"} Nov 02 07:24:00.627749 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:00.627695 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" event={"ID":"3201efd6-0d33-4323-bab0-b76338d090ed","Type":"ContainerStarted","Data":"c005f8658a6a2b5b94bf0e6965c90cd6220a66962ffe7bf0f6e6f1b3ca49461a"} Nov 02 07:24:00.731524 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:00.731479 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" podStartSLOduration=2.917406624 podStartE2EDuration="12.731466211s" totalImagesPullingTime="9.814059587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:00.731012716 +0000 UTC m=+12.883645994" watchObservedRunningTime="2026-11-02 07:24:00.731466211 +0000 UTC m=+12.884099488" Nov 02 07:24:02.100670 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:02.100634 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:02.240007 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:02.239971 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:02.382705 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:02.382607 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:02.382867 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:02.382743 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:02.383139 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:02.383121 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:02.383256 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:02.383231 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:02.383408 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:02.383385 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:02.383502 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:02.383472 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:02.633554 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:02.633471 2577 generic.go:356] "Generic (PLEG): container finished" podID="2eeba619-ec6d-4ec8-828b-2f409d53630d" containerID="55f8cc03b956e487d1c3f597282c23f793a975f837ecf5fdb73fdfa4bff5e92d" exitCode=0 Nov 02 07:24:02.633554 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:02.633528 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkcvh" event={"ID":"2eeba619-ec6d-4ec8-828b-2f409d53630d","Type":"ContainerDied","Data":"55f8cc03b956e487d1c3f597282c23f793a975f837ecf5fdb73fdfa4bff5e92d"} Nov 02 07:24:04.385792 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:04.385635 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:04.386134 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:04.385708 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:04.386134 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:04.385874 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:04.386134 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:04.385940 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:04.386134 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:04.385716 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:04.386134 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:04.386016 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:04.639878 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:04.639781 2577 generic.go:356] "Generic (PLEG): container finished" podID="2eeba619-ec6d-4ec8-828b-2f409d53630d" containerID="07d79a56b18f24da4ad9299ac8c27ba11f3e8a1345afc72e74c66de9d0e1c0da" exitCode=0 Nov 02 07:24:04.639878 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:04.639828 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkcvh" event={"ID":"2eeba619-ec6d-4ec8-828b-2f409d53630d","Type":"ContainerDied","Data":"07d79a56b18f24da4ad9299ac8c27ba11f3e8a1345afc72e74c66de9d0e1c0da"} Nov 02 07:24:04.802145 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:04.802117 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:04.946613 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:04.946573 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:05.112001 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:05.111967 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:05.252796 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:05.252625 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:05.397272 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:05.397238 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:05.644688 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:05.644604 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" event={"ID":"973a9b8f-405e-4dc4-8322-4d12a152178d","Type":"ContainerStarted","Data":"d8bdb1ec2208f6e6f66d77332e14dd7fc9b2d9f57f8f7fc859ce967bfbaa02e7"} Nov 02 07:24:05.644820 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:05.644711 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" event={"ID":"973a9b8f-405e-4dc4-8322-4d12a152178d","Type":"ContainerStarted","Data":"e163d111fe2fd48567a96138e6310f61e4bf7ef823e5173dfa54663caf38ccf3"} Nov 02 07:24:05.644820 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:05.644742 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" event={"ID":"973a9b8f-405e-4dc4-8322-4d12a152178d","Type":"ContainerStarted","Data":"19948733df93bbf10abb2b52c5a270e4c64fa65cafb7ff12ef24b505bb84c6e4"} Nov 02 07:24:05.644820 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:05.644772 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" event={"ID":"973a9b8f-405e-4dc4-8322-4d12a152178d","Type":"ContainerStarted","Data":"2c31e2705f0eaa44987f304778343e220df4c6d464926bc3ab8ca80de3779a5b"} Nov 02 07:24:05.644820 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:05.644803 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" event={"ID":"973a9b8f-405e-4dc4-8322-4d12a152178d","Type":"ContainerStarted","Data":"68f02492f85c40973f71e1408f28f7256bb034d57985461e9ad99b7a95b2276d"} Nov 02 07:24:05.645010 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:05.644829 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" event={"ID":"973a9b8f-405e-4dc4-8322-4d12a152178d","Type":"ContainerStarted","Data":"9ab994834518e26f8352ed996f1eef19e12369df079fb65b269a37a79f073333"} Nov 02 07:24:05.647777 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:05.647753 2577 generic.go:356] "Generic (PLEG): container finished" podID="2eeba619-ec6d-4ec8-828b-2f409d53630d" containerID="e29557fff9cc0f97a93058bc808d2d41ba71765e5ead3562b13dac2ff6cb3204" exitCode=0 Nov 02 07:24:05.647863 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:05.647794 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkcvh" event={"ID":"2eeba619-ec6d-4ec8-828b-2f409d53630d","Type":"ContainerDied","Data":"e29557fff9cc0f97a93058bc808d2d41ba71765e5ead3562b13dac2ff6cb3204"} Nov 02 07:24:06.064087 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:06.063992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:06.064087 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:06.064042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:06.064332 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:06.064151 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:24:06.064332 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:06.064215 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs podName:71998a34-5575-4ae0-8892-2c9e88b1b396 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:22.064202933 +0000 UTC m=+34.216836188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs") pod "network-metrics-daemon-v6hrz" (UID: "71998a34-5575-4ae0-8892-2c9e88b1b396") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:24:06.064332 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:06.064151 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 07:24:06.064332 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:06.064286 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret podName:253a4509-48e5-48c2-8c13-2dc361e5763f nodeName:}" failed. No retries permitted until 2026-11-02 07:24:22.064273631 +0000 UTC m=+34.216906901 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret") pod "global-pull-secret-syncer-9ztwm" (UID: "253a4509-48e5-48c2-8c13-2dc361e5763f") : object "kube-system"/"original-pull-secret" not registered Nov 02 07:24:06.165056 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:06.165009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qfz\" (UniqueName: \"kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz\") pod \"network-check-target-25f6d\" (UID: \"ef891d1c-0c5c-45b3-98eb-563324d8822f\") " pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:06.165239 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:06.165186 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 07:24:06.165239 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:06.165207 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 07:24:06.165239 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:06.165218 2577 projected.go:196] Error preparing data for projected volume kube-api-access-w8qfz for pod openshift-network-diagnostics/network-check-target-25f6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:24:06.165402 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:06.165279 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz podName:ef891d1c-0c5c-45b3-98eb-563324d8822f nodeName:}" failed. No retries permitted until 2026-11-02 07:24:22.165260399 +0000 UTC m=+34.317893660 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w8qfz" (UniqueName: "kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz") pod "network-check-target-25f6d" (UID: "ef891d1c-0c5c-45b3-98eb-563324d8822f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:24:06.385648 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:06.385576 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:06.385770 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:06.385689 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:06.385770 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:06.385705 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:06.385770 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:06.385705 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:06.385877 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:06.385764 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:06.385877 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:06.385846 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:06.406293 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:06.406259 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:06.550064 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:06.550029 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:06.652755 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:06.652680 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkcvh" event={"ID":"2eeba619-ec6d-4ec8-828b-2f409d53630d","Type":"ContainerStarted","Data":"b1aa894012cd26977ed2920a38ec1f97f39cb2a69d45b32c138d3c217bb218b6"} Nov 02 07:24:06.700249 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:06.700203 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qkcvh" podStartSLOduration=8.44748115 podStartE2EDuration="18.700191091s" totalImagesPullingTime="10.252709941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:06.700045229 +0000 UTC m=+18.852678507" watchObservedRunningTime="2026-11-02 07:24:06.700191091 +0000 UTC m=+18.852824369" Nov 02 07:24:06.969674 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:06.969598 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:07.657772 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:07.657687 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" event={"ID":"973a9b8f-405e-4dc4-8322-4d12a152178d","Type":"ContainerStarted","Data":"b04264f6b0597fd49fcbc1876a8acc2e8d2bd8d85c5ba9eed610ef06cfa0e1ab"} Nov 02 07:24:08.383842 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:08.383808 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:08.383989 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:08.383942 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:08.384032 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:08.383988 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:08.384061 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:08.384032 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:08.384126 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:08.384107 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:08.384228 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:08.384208 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:09.569044 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:09.569012 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:09.666304 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:09.666255 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" event={"ID":"973a9b8f-405e-4dc4-8322-4d12a152178d","Type":"ContainerStarted","Data":"59189b89b139e885f11a251ea18b910b8da2c79b213c7df4e792c3b69c739eac"} Nov 02 07:24:09.666530 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:09.666513 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:24:09.681184 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:09.681145 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:24:09.754221 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:09.754120 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" podStartSLOduration=7.351675052 podStartE2EDuration="21.754104714s" totalImagesPullingTime="14.402429662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:09.753533138 +0000 UTC m=+21.906166416" watchObservedRunningTime="2026-11-02 07:24:09.754104714 +0000 UTC m=+21.906737991" Nov 02 07:24:10.382607 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:10.382582 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:10.382607 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:10.382603 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:10.382813 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:10.382589 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:10.382813 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:10.382670 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:10.383194 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:10.383152 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:10.383728 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:10.383695 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:10.668631 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:10.668553 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:24:10.669094 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:10.668633 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:24:10.682794 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:10.682773 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:24:10.856700 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:10.856535 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:10.987175 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:10.987089 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:11.063783 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:11.063759 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f6f9k_fdb3f2d5-5e21-48c3-af74-eb8fd530046a/dns-node-resolver/0.log" Nov 02 07:24:11.191766 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:11.191734 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:11.326866 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:11.326796 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:11.467491 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:11.467460 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:11.774841 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:11.774767 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:11.860660 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:11.860630 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-72ngx_e229a07c-7415-4af9-a9eb-9e7984e0a510/node-ca/0.log" Nov 02 07:24:12.385096 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:12.385063 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:12.385252 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:12.385136 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:12.385252 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:12.385174 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:12.385332 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:12.385251 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:12.385332 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:12.385299 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:12.385440 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:12.385371 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:12.567991 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:12.567960 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:12.976079 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:12.976047 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:13.171419 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:13.171385 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:13.376010 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:13.375912 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:13.661669 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:13.661583 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:13.805787 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:13.805753 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:14.009813 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:14.009738 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:14.142136 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:14.142104 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:14.284647 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:14.284570 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:14.385618 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:14.385591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:14.385750 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:14.385700 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:14.385750 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:14.385712 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:14.385750 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:14.385714 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:14.385875 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:14.385758 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:14.385875 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:14.385812 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:14.480680 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:14.480649 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:14.613953 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:14.613876 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:14.967402 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:14.967325 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:15.177038 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:15.177008 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:15.373739 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:15.373658 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:15.770244 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:15.770154 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:16.385515 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:16.385487 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:16.385928 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:16.385597 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:16.385928 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:16.385619 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:16.385928 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:16.385667 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:16.385928 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:16.385689 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:16.385928 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:16.385775 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:16.571057 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:16.571031 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:16.819556 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:16.819480 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:24:16.819698 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:16.819616 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 07:24:16.820018 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:16.820001 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:24:16.910826 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:16.910795 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:17.052517 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:17.052484 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:17.186040 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:17.185959 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:17.381042 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:17.381008 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:17.761070 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:17.761038 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:18.383304 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:18.383274 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:18.383497 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:18.383397 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:18.383497 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:18.383412 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:18.383497 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:18.383438 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:18.383663 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:18.383503 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:18.383663 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:18.383578 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:18.576533 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:18.576501 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:18.979564 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:18.979538 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:19.137733 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:19.137707 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:19.275344 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:19.275274 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:19.414728 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:19.414696 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:19.775585 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:19.775553 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:20.382886 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:20.382853 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:20.382886 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:20.382859 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:20.383339 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:20.382984 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:20.383339 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:20.382986 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:20.383339 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:20.383078 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:20.383339 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:20.383192 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:20.577137 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:20.577105 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:20.712981 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:20.712903 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:20.972094 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:20.972028 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:21.173588 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:21.173558 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:21.371591 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:21.371516 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:21.511651 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:21.511620 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:21.767591 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:21.767525 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:21.970986 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:21.970961 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:22.090711 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:22.090645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:22.090711 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:22.090677 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:22.090901 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:22.090799 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:24:22.090901 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:22.090805 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 07:24:22.090901 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:22.090849 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs podName:71998a34-5575-4ae0-8892-2c9e88b1b396 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:54.090835298 +0000 UTC m=+66.243468554 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs") pod "network-metrics-daemon-v6hrz" (UID: "71998a34-5575-4ae0-8892-2c9e88b1b396") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 07:24:22.090901 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:22.090863 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret podName:253a4509-48e5-48c2-8c13-2dc361e5763f nodeName:}" failed. No retries permitted until 2026-11-02 07:24:54.090856242 +0000 UTC m=+66.243489498 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret") pod "global-pull-secret-syncer-9ztwm" (UID: "253a4509-48e5-48c2-8c13-2dc361e5763f") : object "kube-system"/"original-pull-secret" not registered Nov 02 07:24:22.191582 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:22.191548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qfz\" (UniqueName: \"kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz\") pod \"network-check-target-25f6d\" (UID: \"ef891d1c-0c5c-45b3-98eb-563324d8822f\") " pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:22.191747 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:22.191675 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 07:24:22.191747 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:22.191690 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 07:24:22.191747 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:22.191700 2577 projected.go:196] Error preparing data for projected volume kube-api-access-w8qfz for pod openshift-network-diagnostics/network-check-target-25f6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:24:22.191854 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:22.191749 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz podName:ef891d1c-0c5c-45b3-98eb-563324d8822f nodeName:}" failed. No retries permitted until 2026-11-02 07:24:54.191737734 +0000 UTC m=+66.344370990 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w8qfz" (UniqueName: "kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz") pod "network-check-target-25f6d" (UID: "ef891d1c-0c5c-45b3-98eb-563324d8822f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 07:24:22.382945 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:22.382868 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:22.383104 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:22.382876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:22.383104 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:22.382975 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:22.383104 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:22.382883 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:22.383104 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:22.383045 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:22.383280 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:22.383108 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:22.397572 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:22.397545 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:22.533747 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:22.533711 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:22.669738 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:22.669654 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:24.380807 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:24.380775 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:24.382874 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:24.382858 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:24.382927 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:24.382880 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:24.382977 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:24.382947 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:24.383013 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:24.382992 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:24.383086 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:24.383070 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:24.383171 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:24.383143 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:24.519398 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:24.519370 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:26.383344 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:26.383303 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:26.383344 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:26.383329 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:26.383891 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:26.383304 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:26.383891 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:26.383430 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:26.383891 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:26.383487 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:26.383891 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:26.383539 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:26.386792 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:26.386770 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:26.542391 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:26.542363 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:26.683774 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:26.683683 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:27.638505 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:27.638421 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9ztwm"] Nov 02 07:24:27.638958 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:27.638586 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:27.638958 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:27.638715 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:27.642607 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:27.642579 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-25f6d"] Nov 02 07:24:27.642770 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:27.642638 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v6hrz"] Nov 02 07:24:27.642770 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:27.642728 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:27.642879 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:27.642831 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:27.643027 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:27.643010 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:27.643107 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:27.643088 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:28.608550 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:28.608515 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:29.030454 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:29.030384 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:29.168172 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:29.168141 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:29.382917 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:29.382848 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:29.382917 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:29.382876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:29.383080 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:29.382943 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:29.383080 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:29.382983 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:29.383080 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:29.383063 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:29.383191 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:29.383122 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:30.589332 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:30.589260 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:30.728289 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:30.728255 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:30.871095 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:30.870884 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:31.383152 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.383122 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:31.383152 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.383149 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:31.383381 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:31.383247 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9ztwm" podUID="253a4509-48e5-48c2-8c13-2dc361e5763f" Nov 02 07:24:31.383381 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:31.383316 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6hrz" podUID="71998a34-5575-4ae0-8892-2c9e88b1b396" Nov 02 07:24:31.383381 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.383339 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:31.383480 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:31.383404 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25f6d" podUID="ef891d1c-0c5c-45b3-98eb-563324d8822f" Nov 02 07:24:31.665136 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.665056 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeReady" Nov 02 07:24:31.665594 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.665216 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 07:24:31.727647 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.727611 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b486r"] Nov 02 07:24:31.729534 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.729513 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.731710 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.731672 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9p24b"] Nov 02 07:24:31.735731 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.735707 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9p24b" Nov 02 07:24:31.745526 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.745503 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kp755"] Nov 02 07:24:31.747537 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.747517 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kp755" Nov 02 07:24:31.747706 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.747682 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 07:24:31.747706 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.747697 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 07:24:31.747706 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.747600 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 07:24:31.747904 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.747723 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jc9nv\"" Nov 02 07:24:31.747904 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.747565 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 07:24:31.754765 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.754449 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b486r"] Nov 02 07:24:31.758773 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.758750 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9p24b"] Nov 02 07:24:31.758960 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.758940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b808d2a-56c8-477f-bca0-d9d69d711516-config-volume\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:24:31.759040 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.758970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e76535e7-a981-4b8a-9bcf-5e400d35c1d1-cert\") pod \"ingress-canary-9p24b\" (UID: \"e76535e7-a981-4b8a-9bcf-5e400d35c1d1\") " pod="openshift-ingress-canary/ingress-canary-9p24b" Nov 02 07:24:31.759040 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.758993 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjrb9\" (UniqueName: \"kubernetes.io/projected/0c592b3d-c373-48f3-a1b4-e75b8134afd6-kube-api-access-qjrb9\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.759209 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.759186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6mt7\" (UniqueName: \"kubernetes.io/projected/1b808d2a-56c8-477f-bca0-d9d69d711516-kube-api-access-d6mt7\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:24:31.759278 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.759241 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b808d2a-56c8-477f-bca0-d9d69d711516-metrics-tls\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:24:31.759278 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.759269 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b808d2a-56c8-477f-bca0-d9d69d711516-tmp-dir\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:24:31.759390 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.759341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zx4h\" (UniqueName: \"kubernetes.io/projected/e76535e7-a981-4b8a-9bcf-5e400d35c1d1-kube-api-access-4zx4h\") pod \"ingress-canary-9p24b\" (UID: \"e76535e7-a981-4b8a-9bcf-5e400d35c1d1\") " pod="openshift-ingress-canary/ingress-canary-9p24b" Nov 02 07:24:31.759390 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.759371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c592b3d-c373-48f3-a1b4-e75b8134afd6-crio-socket\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.759496 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.759395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c592b3d-c373-48f3-a1b4-e75b8134afd6-data-volume\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.759496 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.759452 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c592b3d-c373-48f3-a1b4-e75b8134afd6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.759496 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.759476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c592b3d-c373-48f3-a1b4-e75b8134afd6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.769013 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.768989 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kp755"] Nov 02 07:24:31.848225 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.848194 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 07:24:31.848225 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.848200 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 07:24:31.848436 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.848257 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 07:24:31.848436 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.848300 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 07:24:31.848436 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.848300 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 07:24:31.848436 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.848347 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7x2d2\"" Nov 02 07:24:31.848436 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.848348 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lsjmf\"" Nov 02 07:24:31.859858 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.859835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zx4h\" (UniqueName: \"kubernetes.io/projected/e76535e7-a981-4b8a-9bcf-5e400d35c1d1-kube-api-access-4zx4h\") pod \"ingress-canary-9p24b\" (UID: \"e76535e7-a981-4b8a-9bcf-5e400d35c1d1\") " pod="openshift-ingress-canary/ingress-canary-9p24b" Nov 02 07:24:31.859858 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.859862 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c592b3d-c373-48f3-a1b4-e75b8134afd6-crio-socket\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.860006 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.859875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c592b3d-c373-48f3-a1b4-e75b8134afd6-data-volume\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.860006 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.859898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c592b3d-c373-48f3-a1b4-e75b8134afd6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.860006 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.859916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c592b3d-c373-48f3-a1b4-e75b8134afd6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.860006 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.859951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b808d2a-56c8-477f-bca0-d9d69d711516-config-volume\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:24:31.860006 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.859978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e76535e7-a981-4b8a-9bcf-5e400d35c1d1-cert\") pod \"ingress-canary-9p24b\" (UID: \"e76535e7-a981-4b8a-9bcf-5e400d35c1d1\") " pod="openshift-ingress-canary/ingress-canary-9p24b" Nov 02 07:24:31.860326 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.860010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjrb9\" (UniqueName: \"kubernetes.io/projected/0c592b3d-c373-48f3-a1b4-e75b8134afd6-kube-api-access-qjrb9\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.860326 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.860034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6mt7\" (UniqueName: \"kubernetes.io/projected/1b808d2a-56c8-477f-bca0-d9d69d711516-kube-api-access-d6mt7\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:24:31.860326 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.860061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b808d2a-56c8-477f-bca0-d9d69d711516-metrics-tls\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:24:31.860326 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.860081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b808d2a-56c8-477f-bca0-d9d69d711516-tmp-dir\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:24:31.860326 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.860177 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c592b3d-c373-48f3-a1b4-e75b8134afd6-crio-socket\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.860326 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.860308 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c592b3d-c373-48f3-a1b4-e75b8134afd6-data-volume\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.860612 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.860512 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b808d2a-56c8-477f-bca0-d9d69d711516-tmp-dir\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:24:31.860713 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.860675 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b808d2a-56c8-477f-bca0-d9d69d711516-config-volume\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:24:31.861362 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.861337 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c592b3d-c373-48f3-a1b4-e75b8134afd6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.864028 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.864005 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c592b3d-c373-48f3-a1b4-e75b8134afd6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:31.864110 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.864044 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b808d2a-56c8-477f-bca0-d9d69d711516-metrics-tls\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:24:31.864150 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.864110 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e76535e7-a981-4b8a-9bcf-5e400d35c1d1-cert\") pod \"ingress-canary-9p24b\" (UID: \"e76535e7-a981-4b8a-9bcf-5e400d35c1d1\") " pod="openshift-ingress-canary/ingress-canary-9p24b" Nov 02 07:24:31.882623 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.882598 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6mt7\" (UniqueName: \"kubernetes.io/projected/1b808d2a-56c8-477f-bca0-d9d69d711516-kube-api-access-d6mt7\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:24:31.882720 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.882609 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zx4h\" (UniqueName: \"kubernetes.io/projected/e76535e7-a981-4b8a-9bcf-5e400d35c1d1-kube-api-access-4zx4h\") pod \"ingress-canary-9p24b\" (UID: \"e76535e7-a981-4b8a-9bcf-5e400d35c1d1\") " pod="openshift-ingress-canary/ingress-canary-9p24b" Nov 02 07:24:31.893149 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:31.893129 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjrb9\" (UniqueName: \"kubernetes.io/projected/0c592b3d-c373-48f3-a1b4-e75b8134afd6-kube-api-access-qjrb9\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:32.047279 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:32.047205 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:24:32.049777 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:32.049682 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9p24b" Nov 02 07:24:32.061010 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:32.060990 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kp755" Nov 02 07:24:32.364401 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:32.364329 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9p24b"] Nov 02 07:24:32.376504 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:32.376482 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kp755"] Nov 02 07:24:32.379375 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:32.379352 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b808d2a_56c8_477f_bca0_d9d69d711516.slice/crio-950f78d0d4d456f3acc78856c9ea792bdbe905ec2f628ed1070463c03a3bbc90 WatchSource:0}: Error finding container 950f78d0d4d456f3acc78856c9ea792bdbe905ec2f628ed1070463c03a3bbc90: Status 404 returned error can't find the container with id 950f78d0d4d456f3acc78856c9ea792bdbe905ec2f628ed1070463c03a3bbc90 Nov 02 07:24:32.381578 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:32.381557 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b486r"] Nov 02 07:24:32.385113 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:32.385092 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c592b3d_c373_48f3_a1b4_e75b8134afd6.slice/crio-846c1b684fdfc2efe50a4364b2096eeee11c6f4832421b4f6784cbed929be4b0 WatchSource:0}: Error finding container 846c1b684fdfc2efe50a4364b2096eeee11c6f4832421b4f6784cbed929be4b0: Status 404 returned error can't find the container with id 846c1b684fdfc2efe50a4364b2096eeee11c6f4832421b4f6784cbed929be4b0 Nov 02 07:24:32.712267 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:32.712153 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kp755" event={"ID":"1b808d2a-56c8-477f-bca0-d9d69d711516","Type":"ContainerStarted","Data":"950f78d0d4d456f3acc78856c9ea792bdbe905ec2f628ed1070463c03a3bbc90"} Nov 02 07:24:32.713328 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:32.713303 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9p24b" event={"ID":"e76535e7-a981-4b8a-9bcf-5e400d35c1d1","Type":"ContainerStarted","Data":"2d2977f2c8aa913fba14aa8972796cc2c5a64f43bfaa7f7b2e0393c233d98fd2"} Nov 02 07:24:32.714663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:32.714644 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b486r" event={"ID":"0c592b3d-c373-48f3-a1b4-e75b8134afd6","Type":"ContainerStarted","Data":"a260ce486e1c7191494cef393eb60ae6e71b01d66e9b57fc3980469989bff3b6"} Nov 02 07:24:32.714738 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:32.714713 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b486r" event={"ID":"0c592b3d-c373-48f3-a1b4-e75b8134afd6","Type":"ContainerStarted","Data":"846c1b684fdfc2efe50a4364b2096eeee11c6f4832421b4f6784cbed929be4b0"} Nov 02 07:24:33.382592 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.382509 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:33.382592 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.382539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:33.382820 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.382667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:33.397437 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.397410 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hdb4w\"" Nov 02 07:24:33.409048 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.409020 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 07:24:33.409189 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.409072 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 07:24:33.409380 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.409360 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zwm44\"" Nov 02 07:24:33.409532 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.409517 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhg4q\"" Nov 02 07:24:33.409602 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.409585 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 07:24:33.409663 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.409630 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 07:24:33.522123 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.522083 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:33.656362 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.656281 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:33.720840 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.720793 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b486r" event={"ID":"0c592b3d-c373-48f3-a1b4-e75b8134afd6","Type":"ContainerStarted","Data":"227555b51e8930790362f092132116291e2cb8f67bb62391a621256b94eb1966"} Nov 02 07:24:33.791926 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:33.791854 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:34.725895 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:34.725861 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kp755" event={"ID":"1b808d2a-56c8-477f-bca0-d9d69d711516","Type":"ContainerStarted","Data":"91fdcdf5139024c682c8d0f7ceb102edeacb171ae529d424c3efc6f7f65a9fe3"} Nov 02 07:24:34.726383 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:34.725915 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kp755" event={"ID":"1b808d2a-56c8-477f-bca0-d9d69d711516","Type":"ContainerStarted","Data":"5d0879b7e4cac820ff73091779815151b1f7ef59bccb867fe4d4ce649306d2dd"} Nov 02 07:24:34.726383 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:34.726068 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kp755" Nov 02 07:24:34.728718 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:34.728326 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b486r" event={"ID":"0c592b3d-c373-48f3-a1b4-e75b8134afd6","Type":"ContainerStarted","Data":"6f02b903d76834d4a25e9a402c89b01bcaef77119f37b1c8de538338b491a454"} Nov 02 07:24:34.770233 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:34.770183 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-kp755" podStartSLOduration=1.976723926 podStartE2EDuration="3.770152702s" totalImagesPullingTime="1.793428776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:34.769637031 +0000 UTC m=+46.922270306" watchObservedRunningTime="2026-11-02 07:24:34.770152702 +0000 UTC m=+46.922785979" Nov 02 07:24:34.812489 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:34.812426 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b486r" podStartSLOduration=1.712690104 podStartE2EDuration="3.812411435s" totalImagesPullingTime="2.099721331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:24:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:34.811440489 +0000 UTC m=+46.964073768" watchObservedRunningTime="2026-11-02 07:24:34.812411435 +0000 UTC m=+46.965044712" Nov 02 07:24:35.055215 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.055185 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:35.203983 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.203951 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:35.301808 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.301775 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh"] Nov 02 07:24:35.303988 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.303963 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.319074 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.319052 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 07:24:35.319186 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.319122 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-scd8z\"" Nov 02 07:24:35.319370 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.319354 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 07:24:35.321709 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.321694 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 07:24:35.321791 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.321777 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 07:24:35.321865 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.321853 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 07:24:35.331063 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.331042 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh"] Nov 02 07:24:35.350935 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.350900 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh"] Nov 02 07:24:35.353628 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.353610 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.367573 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.367551 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kk7sz"] Nov 02 07:24:35.367997 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.367978 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:35.369514 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.369498 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.372850 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.372832 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 07:24:35.372969 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.372954 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 07:24:35.377358 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.377337 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh"] Nov 02 07:24:35.386899 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.386877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrwkv\" (UniqueName: \"kubernetes.io/projected/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-api-access-xrwkv\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.386997 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.386918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.386997 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.386940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-tls\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.387061 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387032 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-accelerators-collector-config\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.387093 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387059 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.387093 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0768e3b2-d790-4375-acf5-111424b1c5c3-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.387153 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.387153 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.387236 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387225 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-sys\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.387265 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.387296 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-wtmp\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.387337 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-root\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.387366 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387346 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.387397 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-metrics-client-ca\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.387397 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387379 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.387475 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0768e3b2-d790-4375-acf5-111424b1c5c3-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.387475 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387424 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-textfile\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.387475 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387467 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7xw9\" (UniqueName: \"kubernetes.io/projected/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-kube-api-access-l7xw9\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.387604 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.387491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fk87\" (UniqueName: \"kubernetes.io/projected/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-kube-api-access-8fk87\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.422610 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.422584 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 07:24:35.422771 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.422635 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 07:24:35.422771 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.422651 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-htl44\"" Nov 02 07:24:35.422771 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.422683 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 07:24:35.422771 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.422692 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zwd5q\"" Nov 02 07:24:35.422771 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.422729 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 07:24:35.487894 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.487865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.487894 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.487896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-metrics-client-ca\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.488076 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.487915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.488076 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.487938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0768e3b2-d790-4375-acf5-111424b1c5c3-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.488076 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.487961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-textfile\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.488076 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.487990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7xw9\" (UniqueName: \"kubernetes.io/projected/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-kube-api-access-l7xw9\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.488076 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fk87\" (UniqueName: \"kubernetes.io/projected/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-kube-api-access-8fk87\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.488076 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:35.488034 2577 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Nov 02 07:24:35.488076 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrwkv\" (UniqueName: \"kubernetes.io/projected/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-api-access-xrwkv\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.488423 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.488423 ip-10-0-135-6 kubenswrapper[2577]: E1102 07:24:35.488108 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-tls podName:0768e3b2-d790-4375-acf5-111424b1c5c3 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:35.988089018 +0000 UTC m=+48.140722274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-tls") pod "kube-state-metrics-5dc8d7d89b-sbrnh" (UID: "0768e3b2-d790-4375-acf5-111424b1c5c3") : secret "kube-state-metrics-tls" not found Nov 02 07:24:35.488423 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-tls\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.488423 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-accelerators-collector-config\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.488423 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.488423 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0768e3b2-d790-4375-acf5-111424b1c5c3-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.488423 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488249 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.488808 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.488870 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488798 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.488870 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-sys\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.488969 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488866 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.488969 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.488969 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-wtmp\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.488969 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-root\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.489140 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.488967 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-textfile\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.489140 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.489030 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-root\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.489140 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.489053 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0768e3b2-d790-4375-acf5-111424b1c5c3-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.489140 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.489077 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-sys\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.489140 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.489051 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-accelerators-collector-config\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.489363 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.489148 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0768e3b2-d790-4375-acf5-111424b1c5c3-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.489363 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.489205 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-wtmp\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.489473 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.489453 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-metrics-client-ca\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.491754 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.491728 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.491851 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.491783 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-tls\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.491888 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.491851 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.491922 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.491855 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.491922 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.491911 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.510862 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.510843 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7xw9\" (UniqueName: \"kubernetes.io/projected/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-kube-api-access-l7xw9\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.511042 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.511023 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrwkv\" (UniqueName: \"kubernetes.io/projected/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-api-access-xrwkv\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.511088 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.511055 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fk87\" (UniqueName: \"kubernetes.io/projected/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-kube-api-access-8fk87\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.512912 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.512892 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:35.617688 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.617622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:24:35.648534 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.648505 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:35.682036 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.682009 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:24:35.690681 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:35.690647 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ebc3a2_3bc6_48d7_b78c_f2dd729c4759.slice/crio-e3158881f97bbf37e6370694d1d8e74353afd34e0a3a428d917ce8707f7cd0f8 WatchSource:0}: Error finding container e3158881f97bbf37e6370694d1d8e74353afd34e0a3a428d917ce8707f7cd0f8: Status 404 returned error can't find the container with id e3158881f97bbf37e6370694d1d8e74353afd34e0a3a428d917ce8707f7cd0f8 Nov 02 07:24:35.734558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.734529 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kk7sz" event={"ID":"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759","Type":"ContainerStarted","Data":"e3158881f97bbf37e6370694d1d8e74353afd34e0a3a428d917ce8707f7cd0f8"} Nov 02 07:24:35.735907 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.735868 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9p24b" event={"ID":"e76535e7-a981-4b8a-9bcf-5e400d35c1d1","Type":"ContainerStarted","Data":"13293c2579b2c3d90a4a46c067749fa7a9e864e865e8a8655fd8644709fd94ca"} Nov 02 07:24:35.774387 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.774337 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9p24b" podStartSLOduration=2.06147268 podStartE2EDuration="4.774326012s" totalImagesPullingTime="2.712853332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:35.774121287 +0000 UTC m=+47.926754565" watchObservedRunningTime="2026-11-02 07:24:35.774326012 +0000 UTC m=+47.926959284" Nov 02 07:24:35.796853 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.796826 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:35.922453 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.922384 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh"] Nov 02 07:24:35.925780 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:35.925753 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa67936_9e6b_4f8e_ae72_77f9a41f4dd7.slice/crio-855497f8c5e330e05adc47031d251793180839b97bcdd7d53a2af0bc1a6b53c8 WatchSource:0}: Error finding container 855497f8c5e330e05adc47031d251793180839b97bcdd7d53a2af0bc1a6b53c8: Status 404 returned error can't find the container with id 855497f8c5e330e05adc47031d251793180839b97bcdd7d53a2af0bc1a6b53c8 Nov 02 07:24:35.939103 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.939077 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:35.992794 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.992770 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:35.995089 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:35.995063 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:36.264903 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.264867 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:24:36.572860 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.572777 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh"] Nov 02 07:24:36.574197 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.574136 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:36.576511 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:36.576482 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0768e3b2_d790_4375_acf5_111424b1c5c3.slice/crio-12f8cde28adab95df4cf9180acfce8036a380819d2b0aeaf5da4fad5aa6f92a2 WatchSource:0}: Error finding container 12f8cde28adab95df4cf9180acfce8036a380819d2b0aeaf5da4fad5aa6f92a2: Status 404 returned error can't find the container with id 12f8cde28adab95df4cf9180acfce8036a380819d2b0aeaf5da4fad5aa6f92a2 Nov 02 07:24:36.737370 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.737343 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:36.740704 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.740678 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" event={"ID":"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7","Type":"ContainerStarted","Data":"4804422846caeb79eec6e34a0cd7ded665ae64d36bb67545c9763b8b39df6f81"} Nov 02 07:24:36.740806 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.740784 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" event={"ID":"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7","Type":"ContainerStarted","Data":"5c648775140624087cbfc00fe7ec5ea07a5353949c172b6a0173f62e47221133"} Nov 02 07:24:36.740853 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.740811 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" event={"ID":"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7","Type":"ContainerStarted","Data":"855497f8c5e330e05adc47031d251793180839b97bcdd7d53a2af0bc1a6b53c8"} Nov 02 07:24:36.742373 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.742347 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kk7sz" event={"ID":"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759","Type":"ContainerStarted","Data":"16c3108888d8082597999b12a09c54255fbe301f956ad006067b5e8fa68f0093"} Nov 02 07:24:36.743610 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.743587 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" event={"ID":"0768e3b2-d790-4375-acf5-111424b1c5c3","Type":"ContainerStarted","Data":"12f8cde28adab95df4cf9180acfce8036a380819d2b0aeaf5da4fad5aa6f92a2"} Nov 02 07:24:36.809760 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.809734 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 07:24:36.812039 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.812021 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:36.826934 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.826888 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 07:24:36.827180 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.827143 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 07:24:36.827254 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.827143 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 07:24:36.827254 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.827236 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 07:24:36.827254 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.827248 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 07:24:36.827371 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.827262 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-67pc5\"" Nov 02 07:24:36.827371 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.827289 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 07:24:36.827371 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.827327 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 07:24:36.838805 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.838788 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 07:24:36.839648 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.839634 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 07:24:36.851919 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.851898 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 07:24:36.875540 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.875509 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:36.898704 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.898678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:36.898704 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.898710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-config-volume\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:36.898704 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.898726 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:36.898704 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.898743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:36.899036 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.898768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6mbb\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-kube-api-access-h6mbb\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:36.899036 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.898866 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:36.899036 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.898914 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:36.899036 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.898937 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:36.899036 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.898981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-config-out\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:36.899036 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.899008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-web-config\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:36.899346 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.899042 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:36.899346 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.899110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:36.899346 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:36.899134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000085 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000289 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000289 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-config-volume\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000289 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000289 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000210 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000289 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6mbb\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-kube-api-access-h6mbb\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000289 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000584 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000584 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000336 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000584 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000365 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-config-out\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000584 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-web-config\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000584 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000584 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.000881 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.000702 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.002925 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.001808 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.002925 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.002882 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.003108 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.003080 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-config-out\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.003308 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.003285 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-config-volume\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.003772 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.003722 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.003867 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.003850 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.003932 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.003858 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.003987 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.003924 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.004041 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.003988 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.004943 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.004924 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.005084 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.005068 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-web-config\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.023595 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.023573 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6mbb\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-kube-api-access-h6mbb\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.167837 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.167757 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:24:37.497533 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.497507 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 07:24:37.501395 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:37.501367 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5067c1c4_cfc8_4358_9069_f24576389879.slice/crio-b41ad2171c48e209b8c13466d305258cf3b1051788f44243212abdadebfb1836 WatchSource:0}: Error finding container b41ad2171c48e209b8c13466d305258cf3b1051788f44243212abdadebfb1836: Status 404 returned error can't find the container with id b41ad2171c48e209b8c13466d305258cf3b1051788f44243212abdadebfb1836 Nov 02 07:24:37.616975 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.616942 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6499784568-r4gwp"] Nov 02 07:24:37.623771 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.623747 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.638947 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.638921 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 07:24:37.641741 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.641713 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 07:24:37.641741 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.641733 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 07:24:37.647095 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.647073 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6499784568-r4gwp"] Nov 02 07:24:37.706397 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.706364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ebdbc81-5857-429e-9a08-2143afcdc097-tmp\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.706397 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.706401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.706601 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.706426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-tls\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.706601 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.706528 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ebdbc81-5857-429e-9a08-2143afcdc097-metrics-client-ca\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.706601 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.706556 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-grpc-tls\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.706729 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.706633 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.706729 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.706663 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.706729 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.706693 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.706729 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.706710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mxbc\" (UniqueName: \"kubernetes.io/projected/2ebdbc81-5857-429e-9a08-2143afcdc097-kube-api-access-2mxbc\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.739123 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.739105 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-aovfv7t8ki78k\"" Nov 02 07:24:37.739441 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.739107 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 07:24:37.739441 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.739150 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ln5vb\"" Nov 02 07:24:37.739441 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.739243 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 07:24:37.747496 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.747472 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerStarted","Data":"b41ad2171c48e209b8c13466d305258cf3b1051788f44243212abdadebfb1836"} Nov 02 07:24:37.748802 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.748752 2577 generic.go:356] "Generic (PLEG): container finished" podID="92ebc3a2-3bc6-48d7-b78c-f2dd729c4759" containerID="16c3108888d8082597999b12a09c54255fbe301f956ad006067b5e8fa68f0093" exitCode=0 Nov 02 07:24:37.748895 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.748803 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kk7sz" event={"ID":"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759","Type":"ContainerDied","Data":"16c3108888d8082597999b12a09c54255fbe301f956ad006067b5e8fa68f0093"} Nov 02 07:24:37.808759 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.807747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ebdbc81-5857-429e-9a08-2143afcdc097-tmp\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.808759 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.807782 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.808759 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.807847 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-tls\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.808759 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.807874 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ebdbc81-5857-429e-9a08-2143afcdc097-metrics-client-ca\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.808759 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.807897 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-grpc-tls\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.808759 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.807980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.808759 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.808027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.810626 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.809480 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.810626 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.809529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mxbc\" (UniqueName: \"kubernetes.io/projected/2ebdbc81-5857-429e-9a08-2143afcdc097-kube-api-access-2mxbc\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.813616 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.812971 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.813616 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.813105 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.813616 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.813154 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-grpc-tls\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.813616 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.813469 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ebdbc81-5857-429e-9a08-2143afcdc097-tmp\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.815511 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.814630 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ebdbc81-5857-429e-9a08-2143afcdc097-metrics-client-ca\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.815511 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.814633 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.815511 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.815202 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-tls\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.816104 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.816065 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.834212 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.832790 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mxbc\" (UniqueName: \"kubernetes.io/projected/2ebdbc81-5857-429e-9a08-2143afcdc097-kube-api-access-2mxbc\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:37.940377 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:37.940142 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:38.176378 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.176341 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:38.256073 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.255991 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6499784568-r4gwp"] Nov 02 07:24:38.259763 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:38.259734 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ebdbc81_5857_429e_9a08_2143afcdc097.slice/crio-c1bdf19a73e7982a74f1c4ffc53fc327d44336729db358619c1a892ec8604367 WatchSource:0}: Error finding container c1bdf19a73e7982a74f1c4ffc53fc327d44336729db358619c1a892ec8604367: Status 404 returned error can't find the container with id c1bdf19a73e7982a74f1c4ffc53fc327d44336729db358619c1a892ec8604367 Nov 02 07:24:38.314076 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.314036 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:38.599276 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.599248 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:38.729132 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.729097 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:38.757204 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.757074 2577 generic.go:356] "Generic (PLEG): container finished" podID="5067c1c4-cfc8-4358-9069-f24576389879" containerID="aa3e3d153f3a65cffc261804057dc9dbc532f39e6c0324199097adc4aea6bd2c" exitCode=0 Nov 02 07:24:38.757204 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.757189 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerDied","Data":"aa3e3d153f3a65cffc261804057dc9dbc532f39e6c0324199097adc4aea6bd2c"} Nov 02 07:24:38.759586 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.759505 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kk7sz" event={"ID":"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759","Type":"ContainerStarted","Data":"e08c62b95355708c8f291e845bc02c5d5fe6564d7af4f28f29604f3e51bb1cbc"} Nov 02 07:24:38.759586 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.759554 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kk7sz" event={"ID":"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759","Type":"ContainerStarted","Data":"3fe30df70ab413322c56b8c8a09e5615f12f2487ef8d0b23b0d40c0ba1edb824"} Nov 02 07:24:38.762526 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.762503 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" event={"ID":"0768e3b2-d790-4375-acf5-111424b1c5c3","Type":"ContainerStarted","Data":"dddd0025b8b2af2af20544a03794a4a7721b9069b1c23e90c22c2d93139c50f8"} Nov 02 07:24:38.762635 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.762562 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" event={"ID":"0768e3b2-d790-4375-acf5-111424b1c5c3","Type":"ContainerStarted","Data":"c66dce90335b7a4401f5166cc4f37106cf1a621a86d7b4a57db5462b164f81ff"} Nov 02 07:24:38.762635 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.762593 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" event={"ID":"0768e3b2-d790-4375-acf5-111424b1c5c3","Type":"ContainerStarted","Data":"5cb5a108b6926917aadd46428d1c3101207759069864ff9905b68c1d4da29e1e"} Nov 02 07:24:38.763798 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.763777 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" event={"ID":"2ebdbc81-5857-429e-9a08-2143afcdc097","Type":"ContainerStarted","Data":"c1bdf19a73e7982a74f1c4ffc53fc327d44336729db358619c1a892ec8604367"} Nov 02 07:24:38.852259 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.852153 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kk7sz" podStartSLOduration=2.828224786 podStartE2EDuration="3.852134736s" totalImagesPullingTime="957.591766ms" totalInitContainerRuntime="66.318184ms" isStatefulPod=true podCreationTimestamp="2026-11-02 07:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:38.851462063 +0000 UTC m=+51.004095352" watchObservedRunningTime="2026-11-02 07:24:38.852134736 +0000 UTC m=+51.004768015" Nov 02 07:24:38.897176 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:38.895695 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" podStartSLOduration=2.748926752 podStartE2EDuration="3.895677027s" totalImagesPullingTime="1.146750275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:38.895522576 +0000 UTC m=+51.048155871" watchObservedRunningTime="2026-11-02 07:24:38.895677027 +0000 UTC m=+51.048310306" Nov 02 07:24:39.771517 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.771479 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" event={"ID":"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7","Type":"ContainerStarted","Data":"7f84fc49842f5eaea5be0f43f06fd8e7fde9076c2ed4fb4c0958bcab6c789618"} Nov 02 07:24:39.814708 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.814651 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" podStartSLOduration=1.952475535 podStartE2EDuration="4.814636703s" totalImagesPullingTime="2.862161168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:39.814038869 +0000 UTC m=+51.966672149" watchObservedRunningTime="2026-11-02 07:24:39.814636703 +0000 UTC m=+51.967269982" Nov 02 07:24:39.827766 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.827730 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-895bcdc89-gxp7k"] Nov 02 07:24:39.831464 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.831441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:39.846319 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.846259 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-895bcdc89-gxp7k"] Nov 02 07:24:39.855522 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.855215 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 07:24:39.855522 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.855247 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 07:24:39.857104 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.857083 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 07:24:39.857228 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.857107 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 07:24:39.857228 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.857188 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rhkvn\"" Nov 02 07:24:39.857312 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.857290 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7n5qmumkac75f\"" Nov 02 07:24:39.925010 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.924975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-client-ca-bundle\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:39.925193 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.925086 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hrz7\" (UniqueName: \"kubernetes.io/projected/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-kube-api-access-5hrz7\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:39.925193 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.925183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-metrics-server-audit-profiles\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:39.925296 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.925213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-audit-log\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:39.925296 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.925248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-secret-metrics-server-client-certs\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:39.925296 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.925285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-secret-metrics-server-tls\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:39.925392 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:39.925317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.026252 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.026177 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-client-ca-bundle\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.026252 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.026238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hrz7\" (UniqueName: \"kubernetes.io/projected/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-kube-api-access-5hrz7\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.026461 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.026288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-metrics-server-audit-profiles\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.026461 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.026318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-audit-log\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.026461 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.026358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-secret-metrics-server-client-certs\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.026461 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.026400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-secret-metrics-server-tls\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.026461 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.026425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.026852 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.026822 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-audit-log\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.027746 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.027716 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.027859 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.027815 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-metrics-server-audit-profiles\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.029411 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.029368 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-client-ca-bundle\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.030018 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.029992 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-secret-metrics-server-client-certs\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.030127 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.030095 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-secret-metrics-server-tls\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.049700 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.049674 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hrz7\" (UniqueName: \"kubernetes.io/projected/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-kube-api-access-5hrz7\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.146204 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.145972 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:24:40.152347 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.152281 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:40.162527 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.162504 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l"] Nov 02 07:24:40.166058 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.166041 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:24:40.180463 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.180423 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 07:24:40.180707 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.180685 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tn89g\"" Nov 02 07:24:40.187282 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.187257 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l"] Nov 02 07:24:40.228606 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.228574 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xfn8l\" (UID: \"b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:24:40.228766 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.228746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6nkn\" (UniqueName: \"kubernetes.io/projected/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0-kube-api-access-d6nkn\") pod \"monitoring-plugin-77f4f76594-xfn8l\" (UID: \"b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:24:40.286620 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.286555 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:40.329811 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.329788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xfn8l\" (UID: \"b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:24:40.329944 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.329892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6nkn\" (UniqueName: \"kubernetes.io/projected/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0-kube-api-access-d6nkn\") pod \"monitoring-plugin-77f4f76594-xfn8l\" (UID: \"b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:24:40.335766 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.335740 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xfn8l\" (UID: \"b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:24:40.353086 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.353064 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6nkn\" (UniqueName: \"kubernetes.io/projected/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0-kube-api-access-d6nkn\") pod \"monitoring-plugin-77f4f76594-xfn8l\" (UID: \"b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:24:40.419697 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.419303 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:40.471370 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.471345 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-895bcdc89-gxp7k"] Nov 02 07:24:40.481861 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.481604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:24:40.483639 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:40.483601 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e577f03_1ea2_4496_9adb_84a2dc4dc72f.slice/crio-59e951ef66835dbeebaa695e657955359cb030eff48402f0d5dbe9463ba0be46 WatchSource:0}: Error finding container 59e951ef66835dbeebaa695e657955359cb030eff48402f0d5dbe9463ba0be46: Status 404 returned error can't find the container with id 59e951ef66835dbeebaa695e657955359cb030eff48402f0d5dbe9463ba0be46 Nov 02 07:24:40.777198 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.777140 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" event={"ID":"2ebdbc81-5857-429e-9a08-2143afcdc097","Type":"ContainerStarted","Data":"03789706e2a1256726a92c48c6248edbc97b7e17c6dfd93a45883216a2a8f36e"} Nov 02 07:24:40.777646 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.777242 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" event={"ID":"2ebdbc81-5857-429e-9a08-2143afcdc097","Type":"ContainerStarted","Data":"6e95e178092352bd7f0bf9588068e3fa6f5b01c10684f696abed0c94a347e757"} Nov 02 07:24:40.777646 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.777285 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" event={"ID":"2ebdbc81-5857-429e-9a08-2143afcdc097","Type":"ContainerStarted","Data":"3dd2a5589c9f1a147ea7ad9825995799ef02d979711e21adda5338797d4a5b85"} Nov 02 07:24:40.777646 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.777363 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:40.780205 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.780178 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerStarted","Data":"d108b1ba1274ac67203c71a5c6b5133fdf8402b0707d3229b48bba80d503ddcd"} Nov 02 07:24:40.780290 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.780230 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerStarted","Data":"0f7556c644d1625c07cfb8660f8d413bb604057329cb98f66b9c3f01007c1efc"} Nov 02 07:24:40.780290 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.780270 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerStarted","Data":"27128c6aee92197baab9a76c99d441338d6bbdde3c3f2cc99f4fc5493d6f8462"} Nov 02 07:24:40.780367 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.780307 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerStarted","Data":"7dc06196250deaccc91c194e66dc0d821d6d9ec5977a08579c9fae4e5afc2d5a"} Nov 02 07:24:40.780367 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.780341 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerStarted","Data":"f63e67969306d93adfebea905e48f96b10b8be605fa957d8bebded6f85c556d0"} Nov 02 07:24:40.781298 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.781278 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" event={"ID":"0e577f03-1ea2-4496-9adb-84a2dc4dc72f","Type":"ContainerStarted","Data":"59e951ef66835dbeebaa695e657955359cb030eff48402f0d5dbe9463ba0be46"} Nov 02 07:24:40.784974 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.784956 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l"] Nov 02 07:24:40.788294 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:40.788275 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9aa31c8_4e1b_4fb5_b26a_ab0089e364c0.slice/crio-332528e1ae9abba2f8985c8bdde36e38f596b5208f97d9ed3292612c129390b6 WatchSource:0}: Error finding container 332528e1ae9abba2f8985c8bdde36e38f596b5208f97d9ed3292612c129390b6: Status 404 returned error can't find the container with id 332528e1ae9abba2f8985c8bdde36e38f596b5208f97d9ed3292612c129390b6 Nov 02 07:24:40.918051 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:40.918017 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:41.788382 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:41.788344 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" event={"ID":"2ebdbc81-5857-429e-9a08-2143afcdc097","Type":"ContainerStarted","Data":"b7d1040f56706f4d3391a5c5947de2806043dd2215225fe1151532187845293c"} Nov 02 07:24:41.788865 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:41.788430 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" event={"ID":"2ebdbc81-5857-429e-9a08-2143afcdc097","Type":"ContainerStarted","Data":"64ffe9ab4eadc1c068d3d75b385083101b08bbd64e41fab0b1025b4876603dd0"} Nov 02 07:24:41.788865 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:41.788476 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" event={"ID":"2ebdbc81-5857-429e-9a08-2143afcdc097","Type":"ContainerStarted","Data":"6a98b47a21b4709ce99f1d724f731c59f232599d556c199536302ca8989dd1d1"} Nov 02 07:24:41.788865 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:41.788522 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:41.792100 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:41.792072 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerStarted","Data":"e7d6f87053a8f2d472dd83cc9d778f0bd1bfb30652c9452f693cbbcfade68c5d"} Nov 02 07:24:41.793327 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:41.793291 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" event={"ID":"b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0","Type":"ContainerStarted","Data":"332528e1ae9abba2f8985c8bdde36e38f596b5208f97d9ed3292612c129390b6"} Nov 02 07:24:41.867747 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:41.867687 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" podStartSLOduration=1.8493601769999999 podStartE2EDuration="4.867670674s" totalImagesPullingTime="3.018310497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:41.85449557 +0000 UTC m=+54.007128859" watchObservedRunningTime="2026-11-02 07:24:41.867670674 +0000 UTC m=+54.020303953" Nov 02 07:24:41.916234 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:41.916173 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.538085895 podStartE2EDuration="5.91614271s" totalImagesPullingTime="3.364670421s" totalInitContainerRuntime="13.386394ms" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:41.914794615 +0000 UTC m=+54.067427893" watchObservedRunningTime="2026-11-02 07:24:41.91614271 +0000 UTC m=+54.068775992" Nov 02 07:24:42.166452 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.166418 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cc9dd686b-z95lj"] Nov 02 07:24:42.169677 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.169650 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.187648 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.187607 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 07:24:42.187780 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.187672 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 07:24:42.187780 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.187672 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 07:24:42.187780 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.187691 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 07:24:42.200199 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.200144 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 07:24:42.200358 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.200301 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 07:24:42.201497 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.201474 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cc9dd686b-z95lj"] Nov 02 07:24:42.211454 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.211345 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 07:24:42.211454 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.211420 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jcgqg\"" Nov 02 07:24:42.211766 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.211741 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 07:24:42.247314 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.247283 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-trusted-ca-bundle\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.247474 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.247341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-serving-cert\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.247474 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.247367 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-tmp\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.247474 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.247416 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbk9t\" (UniqueName: \"kubernetes.io/projected/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-kube-api-access-dbk9t\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.247474 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.247438 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-oauth-serving-cert\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.247656 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.247474 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-config\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.247656 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.247495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-oauth-config\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.247656 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.247523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-service-ca\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.271190 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.271143 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 07:24:42.275577 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.275553 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.295926 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.295854 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 07:24:42.295926 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.295874 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 07:24:42.295926 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.295903 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 07:24:42.297029 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.297010 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 07:24:42.308342 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.308316 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 07:24:42.310538 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.310445 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 07:24:42.311734 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.311713 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 07:24:42.311843 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.311771 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 07:24:42.311843 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.311798 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 07:24:42.311843 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.311838 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 07:24:42.311993 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.311883 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 07:24:42.311993 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.311903 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 07:24:42.312089 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.312011 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 07:24:42.312089 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.312032 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-45mkt\"" Nov 02 07:24:42.312089 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.312057 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-32bjlgfpvos9j\"" Nov 02 07:24:42.312089 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.312085 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 07:24:42.312278 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.312099 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 07:24:42.315543 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.315293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:42.348039 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-serving-cert\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.348186 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348186 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348186 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-tmp\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.348186 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4hv6\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-kube-api-access-n4hv6\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348186 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348444 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbk9t\" (UniqueName: \"kubernetes.io/projected/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-kube-api-access-dbk9t\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.348444 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-oauth-serving-cert\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.348444 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348444 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348403 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348444 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348422 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348596 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-config\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.348596 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348463 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-oauth-config\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.348596 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-web-config\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348596 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348596 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348525 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-tmp\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.348596 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-service-ca\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.348596 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config-out\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348858 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348594 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348858 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348858 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348639 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348858 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348667 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348858 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348858 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348858 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348770 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.348858 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348797 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-trusted-ca-bundle\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.348858 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348818 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.349311 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.349311 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.348923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.349311 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.349072 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-oauth-serving-cert\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.349460 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.349314 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-service-ca\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.349825 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.349802 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-config\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.350180 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.350134 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-trusted-ca-bundle\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.351676 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.351652 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-oauth-config\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.351829 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.351811 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-serving-cert\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.372135 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.372115 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbk9t\" (UniqueName: \"kubernetes.io/projected/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-kube-api-access-dbk9t\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.449704 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.449674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.449704 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.449709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.449935 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.449729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.449935 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.449766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-web-config\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.449935 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.449794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.449935 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.449822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config-out\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.449935 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.449845 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.449935 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.449870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.449935 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.449895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.449935 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.449921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.450508 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.449958 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.450508 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.450002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.450508 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.450025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.450508 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.450050 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.450508 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.450078 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.450508 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.450102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.450794 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.450617 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.450794 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.450710 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.450794 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.450744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.450794 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.450780 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hv6\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-kube-api-access-n4hv6\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.450989 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.450809 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.453628 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.451557 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.453628 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.452011 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.453628 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.453091 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.453628 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.453286 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.453628 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.453415 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.454671 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.454631 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.454758 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.454701 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.455209 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.455188 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.455825 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.455802 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.456399 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.456380 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.457012 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.456992 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.461631 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.457973 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.461631 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.458277 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config-out\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.461631 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.458718 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-web-config\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.461631 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.458839 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.461631 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.458944 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.461631 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.459105 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.461631 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.459921 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.478977 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.478956 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hv6\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-kube-api-access-n4hv6\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.482828 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.482807 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:42.604417 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.604388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:42.683875 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.683849 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:24:42.800387 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.800342 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" event={"ID":"b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0","Type":"ContainerStarted","Data":"a84cae1865fa5fa7a87f76475dcfab1fda55e145e1549609cc9dcd001e1a5099"} Nov 02 07:24:42.800800 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.800405 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:24:42.801873 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.801835 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" event={"ID":"0e577f03-1ea2-4496-9adb-84a2dc4dc72f","Type":"ContainerStarted","Data":"51f18bbea357bbe84636ed24775fac37a15946f3aa66c8959f48418e752490f9"} Nov 02 07:24:42.805558 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.805536 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:24:42.808284 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.808259 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cc9dd686b-z95lj"] Nov 02 07:24:42.810774 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:42.810744 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bcb2cc_73a5_4d9e_b0db_aaf5f3666c02.slice/crio-792a2c51328c1027a6c630f5c72143d25ecd4946240c1adf01746da581a9a899 WatchSource:0}: Error finding container 792a2c51328c1027a6c630f5c72143d25ecd4946240c1adf01746da581a9a899: Status 404 returned error can't find the container with id 792a2c51328c1027a6c630f5c72143d25ecd4946240c1adf01746da581a9a899 Nov 02 07:24:42.839112 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.839045 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" podStartSLOduration=1.108942471 podStartE2EDuration="2.839034403s" totalImagesPullingTime="1.730091932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:42.838887878 +0000 UTC m=+54.991521156" watchObservedRunningTime="2026-11-02 07:24:42.839034403 +0000 UTC m=+54.991667672" Nov 02 07:24:42.881913 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.881865 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" podStartSLOduration=2.302278427 podStartE2EDuration="3.88185106s" totalImagesPullingTime="1.579572633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:42.88052141 +0000 UTC m=+55.033154684" watchObservedRunningTime="2026-11-02 07:24:42.88185106 +0000 UTC m=+55.034484337" Nov 02 07:24:42.923151 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:42.923129 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 07:24:42.925590 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:42.925563 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e5522b_3924_4cc4_8e6a_34e7e7c2e0e8.slice/crio-92d751fe6bb0ba81d295c92ae8ca43233805fd440339cad982dd6fc8c293ad43 WatchSource:0}: Error finding container 92d751fe6bb0ba81d295c92ae8ca43233805fd440339cad982dd6fc8c293ad43: Status 404 returned error can't find the container with id 92d751fe6bb0ba81d295c92ae8ca43233805fd440339cad982dd6fc8c293ad43 Nov 02 07:24:43.807769 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:43.807729 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc9dd686b-z95lj" event={"ID":"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02","Type":"ContainerStarted","Data":"792a2c51328c1027a6c630f5c72143d25ecd4946240c1adf01746da581a9a899"} Nov 02 07:24:43.809531 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:43.809507 2577 generic.go:356] "Generic (PLEG): container finished" podID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerID="4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a" exitCode=0 Nov 02 07:24:43.809647 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:43.809594 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerDied","Data":"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a"} Nov 02 07:24:43.809705 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:43.809666 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerStarted","Data":"92d751fe6bb0ba81d295c92ae8ca43233805fd440339cad982dd6fc8c293ad43"} Nov 02 07:24:44.015621 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:44.015584 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:44.151014 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:44.150981 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:44.738578 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:44.738542 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kp755" Nov 02 07:24:44.969894 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:44.969855 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:45.108036 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:45.107963 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:45.820836 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:45.820792 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc9dd686b-z95lj" event={"ID":"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02","Type":"ContainerStarted","Data":"3beb96bb221886bd05c82064c126d086524a4b1a10211337234edb6c061e1fc0"} Nov 02 07:24:45.863962 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:45.863906 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7cc9dd686b-z95lj" podStartSLOduration=1.043808973 podStartE2EDuration="3.863889763s" totalImagesPullingTime="2.82008079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:45.862505241 +0000 UTC m=+58.015138522" watchObservedRunningTime="2026-11-02 07:24:45.863889763 +0000 UTC m=+58.016523040" Nov 02 07:24:46.214533 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:46.214498 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:46.358059 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:46.358010 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:47.808444 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:47.808419 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:24:47.834345 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:47.834312 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerStarted","Data":"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd"} Nov 02 07:24:47.834472 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:47.834395 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerStarted","Data":"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c"} Nov 02 07:24:47.834472 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:47.834425 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerStarted","Data":"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5"} Nov 02 07:24:47.834472 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:47.834454 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerStarted","Data":"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62"} Nov 02 07:24:47.834651 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:47.834477 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerStarted","Data":"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc"} Nov 02 07:24:47.834651 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:47.834500 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerStarted","Data":"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3"} Nov 02 07:24:47.925319 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:47.925273 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.907561235 podStartE2EDuration="5.925259202s" totalImagesPullingTime="3.00180516s" totalInitContainerRuntime="15.892807ms" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:47.923661135 +0000 UTC m=+60.076294413" watchObservedRunningTime="2026-11-02 07:24:47.925259202 +0000 UTC m=+60.077892479" Nov 02 07:24:48.398882 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:48.398853 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:48.542960 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:48.542918 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:48.722369 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:48.722297 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:48.864943 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:48.864907 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:52.483178 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:52.483138 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:52.483630 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:52.483206 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:52.487794 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:52.487773 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:52.605417 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:52.605383 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:24:52.857752 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:52.857680 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:24:54.160660 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.160622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:54.161077 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.160692 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:54.185807 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.185779 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 07:24:54.185807 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.185805 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 07:24:54.193031 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.193007 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:54.193137 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.193047 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:54.261450 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.261416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qfz\" (UniqueName: \"kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz\") pod \"network-check-target-25f6d\" (UID: \"ef891d1c-0c5c-45b3-98eb-563324d8822f\") " pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:54.275142 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.275116 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 07:24:54.295500 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.295476 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 07:24:54.304846 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.304817 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8qfz\" (UniqueName: \"kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz\") pod \"network-check-target-25f6d\" (UID: \"ef891d1c-0c5c-45b3-98eb-563324d8822f\") " pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:54.414264 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.414186 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hdb4w\"" Nov 02 07:24:54.421429 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.421408 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:54.421972 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.421956 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zwm44\"" Nov 02 07:24:54.427335 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.427323 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhg4q\"" Nov 02 07:24:54.429438 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.429418 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:24:54.434436 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.434420 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:24:54.727383 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.727359 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-25f6d"] Nov 02 07:24:54.730411 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:54.730379 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef891d1c_0c5c_45b3_98eb_563324d8822f.slice/crio-3b2ac46dfbc30ba5a825a838554b0c6d6a8a73dc25ec2ddfebe032b9eb260c97 WatchSource:0}: Error finding container 3b2ac46dfbc30ba5a825a838554b0c6d6a8a73dc25ec2ddfebe032b9eb260c97: Status 404 returned error can't find the container with id 3b2ac46dfbc30ba5a825a838554b0c6d6a8a73dc25ec2ddfebe032b9eb260c97 Nov 02 07:24:54.738119 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.738096 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9ztwm"] Nov 02 07:24:54.741096 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:54.741065 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253a4509_48e5_48c2_8c13_2dc361e5763f.slice/crio-8df284fb36e6a77a897fa231542dcdbde2bcce714111338c842b39b67a0e4a72 WatchSource:0}: Error finding container 8df284fb36e6a77a897fa231542dcdbde2bcce714111338c842b39b67a0e4a72: Status 404 returned error can't find the container with id 8df284fb36e6a77a897fa231542dcdbde2bcce714111338c842b39b67a0e4a72 Nov 02 07:24:54.741471 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.741338 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v6hrz"] Nov 02 07:24:54.743759 ip-10-0-135-6 kubenswrapper[2577]: W1102 07:24:54.743708 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71998a34_5575_4ae0_8892_2c9e88b1b396.slice/crio-3547e0dafded1783e257bc3b835241e079ca4b199bbd1167bf4d7dc2939701e6 WatchSource:0}: Error finding container 3547e0dafded1783e257bc3b835241e079ca4b199bbd1167bf4d7dc2939701e6: Status 404 returned error can't find the container with id 3547e0dafded1783e257bc3b835241e079ca4b199bbd1167bf4d7dc2939701e6 Nov 02 07:24:54.861547 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.861514 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9ztwm" event={"ID":"253a4509-48e5-48c2-8c13-2dc361e5763f","Type":"ContainerStarted","Data":"8df284fb36e6a77a897fa231542dcdbde2bcce714111338c842b39b67a0e4a72"} Nov 02 07:24:54.862529 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.862506 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-25f6d" event={"ID":"ef891d1c-0c5c-45b3-98eb-563324d8822f","Type":"ContainerStarted","Data":"3b2ac46dfbc30ba5a825a838554b0c6d6a8a73dc25ec2ddfebe032b9eb260c97"} Nov 02 07:24:54.863525 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:54.863499 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v6hrz" event={"ID":"71998a34-5575-4ae0-8892-2c9e88b1b396","Type":"ContainerStarted","Data":"3547e0dafded1783e257bc3b835241e079ca4b199bbd1167bf4d7dc2939701e6"} Nov 02 07:24:55.119903 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:55.119822 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:55.264300 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:55.264263 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:55.984098 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:55.983079 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:56.153106 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:56.153021 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:56.873313 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:56.873280 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v6hrz" event={"ID":"71998a34-5575-4ae0-8892-2c9e88b1b396","Type":"ContainerStarted","Data":"9445e7c7c7493c4a9a69906304604dc8a855111d74ba1052985cbce7e7bb760b"} Nov 02 07:24:56.873658 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:56.873331 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v6hrz" event={"ID":"71998a34-5575-4ae0-8892-2c9e88b1b396","Type":"ContainerStarted","Data":"00dc3e4064d1c7d9d7ea6fcbc96a349f27ec7fa297699a1b0e341284bf32d268"} Nov 02 07:24:56.912716 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:56.912661 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v6hrz" podStartSLOduration=67.81868091 podStartE2EDuration="1m8.912641321s" totalImagesPullingTime="1.093960411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:56.912433067 +0000 UTC m=+69.065066346" watchObservedRunningTime="2026-11-02 07:24:56.912641321 +0000 UTC m=+69.065274601" Nov 02 07:24:58.885568 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:58.885522 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-25f6d" event={"ID":"ef891d1c-0c5c-45b3-98eb-563324d8822f","Type":"ContainerStarted","Data":"2cbdb73f9bb02b3b141c06867c48bbba8dfb191ed6e152b60f2c84629515532a"} Nov 02 07:24:58.886017 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:58.885674 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:24:58.925766 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:58.925711 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-25f6d" podStartSLOduration=67.66115132 podStartE2EDuration="1m10.925692639s" totalImagesPullingTime="3.264541319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:58.923762253 +0000 UTC m=+71.076395545" watchObservedRunningTime="2026-11-02 07:24:58.925692639 +0000 UTC m=+71.078325919" Nov 02 07:24:59.509151 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:59.509115 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:59.642317 ip-10-0-135-6 kubenswrapper[2577]: I1102 07:24:59.642190 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 07:24:59.642342 ip-10-0-135-6 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 07:24:59.660571 ip-10-0-135-6 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 07:24:59.660737 ip-10-0-135-6 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 07:24:59.660896 ip-10-0-135-6 systemd[1]: kubelet.service: Consumed 4.681s CPU time. Nov 02 07:24:59.669800 ip-10-0-135-6 systemd[1]: Starting Kubernetes Kubelet... Nov 02 07:24:59.730822 ip-10-0-135-6 kubenswrapper[5899]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 07:24:59.730822 ip-10-0-135-6 kubenswrapper[5899]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 07:24:59.730822 ip-10-0-135-6 kubenswrapper[5899]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 07:24:59.730822 ip-10-0-135-6 kubenswrapper[5899]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 07:24:59.732839 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732765 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:24:59.732839 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732838 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:24:59.732839 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732843 5899 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732846 5899 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732849 5899 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732852 5899 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732854 5899 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732857 5899 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732860 5899 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732863 5899 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732866 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732869 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732871 5899 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732874 5899 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732877 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732880 5899 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732882 5899 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732885 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732888 5899 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732890 5899 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732893 5899 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:24:59.732945 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732896 5899 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732899 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732902 5899 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732905 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732907 5899 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732910 5899 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732913 5899 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732915 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732918 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732921 5899 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732923 5899 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732926 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732929 5899 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732932 5899 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732934 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732937 5899 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732940 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732943 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732946 5899 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:24:59.733439 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732949 5899 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732952 5899 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732955 5899 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732958 5899 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732961 5899 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732964 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732967 5899 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732969 5899 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732972 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732975 5899 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732977 5899 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732980 5899 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732982 5899 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732985 5899 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732987 5899 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732990 5899 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732993 5899 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732995 5899 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.732997 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733000 5899 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:24:59.733912 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733003 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733005 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733008 5899 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733010 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733013 5899 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733020 5899 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733023 5899 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733027 5899 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733032 5899 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733035 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733038 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733041 5899 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733043 5899 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733047 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733051 5899 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733055 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733058 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733060 5899 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733063 5899 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:24:59.734416 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733066 5899 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733069 5899 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733071 5899 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733074 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733076 5899 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733079 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733082 5899 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733084 5899 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733087 5899 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733089 5899 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733092 5899 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733094 5899 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733097 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733099 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733102 5899 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733105 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733108 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733111 5899 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733113 5899 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733117 5899 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:24:59.734900 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733119 5899 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733122 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733125 5899 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733127 5899 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733130 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733132 5899 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733136 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733904 5899 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733912 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733915 5899 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733918 5899 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733921 5899 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733924 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733927 5899 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733930 5899 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733933 5899 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733936 5899 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733939 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733941 5899 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733944 5899 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:24:59.735402 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733946 5899 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733949 5899 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733952 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733954 5899 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733957 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733960 5899 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733962 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733965 5899 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733967 5899 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733970 5899 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733973 5899 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733976 5899 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733984 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733987 5899 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733990 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733992 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733995 5899 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.733997 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734001 5899 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:24:59.735897 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734003 5899 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734006 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734009 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734011 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734014 5899 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734016 5899 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734018 5899 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734021 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734024 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734026 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734029 5899 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734031 5899 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734034 5899 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734036 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734039 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734041 5899 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734044 5899 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734046 5899 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734049 5899 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:24:59.736390 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734051 5899 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734054 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734056 5899 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734059 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734061 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734066 5899 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734069 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734077 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734080 5899 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734083 5899 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734085 5899 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734088 5899 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734091 5899 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734093 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734096 5899 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734098 5899 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734101 5899 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734104 5899 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734107 5899 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734111 5899 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:24:59.736864 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734116 5899 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734119 5899 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734122 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734125 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734129 5899 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734132 5899 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734134 5899 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734137 5899 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734140 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734142 5899 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734145 5899 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734147 5899 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734150 5899 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734153 5899 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734155 5899 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734173 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734178 5899 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734183 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734186 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734190 5899 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:24:59.737379 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734193 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734196 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734199 5899 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734201 5899 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734204 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734207 5899 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734209 5899 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734212 5899 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734215 5899 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734217 5899 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734220 5899 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734222 5899 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734225 5899 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734229 5899 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734233 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734722 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734729 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734732 5899 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734734 5899 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:24:59.737891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734737 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734740 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734743 5899 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734746 5899 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734748 5899 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734751 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734754 5899 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734756 5899 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734758 5899 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734761 5899 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734765 5899 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734767 5899 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734772 5899 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734776 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734779 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734781 5899 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734784 5899 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734786 5899 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734789 5899 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734791 5899 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:24:59.738397 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734794 5899 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734797 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734799 5899 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734802 5899 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734804 5899 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734807 5899 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734809 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734812 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734814 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734817 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734819 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734822 5899 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734824 5899 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734827 5899 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734829 5899 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734832 5899 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734835 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734837 5899 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734839 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:24:59.739252 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734842 5899 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734844 5899 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734847 5899 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734850 5899 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734853 5899 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734856 5899 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734859 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734862 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734865 5899 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734867 5899 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734870 5899 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734874 5899 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734877 5899 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734879 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734882 5899 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734884 5899 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734887 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734889 5899 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734892 5899 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:24:59.740023 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734895 5899 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734898 5899 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734900 5899 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734903 5899 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734905 5899 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734908 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734911 5899 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734913 5899 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734916 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734918 5899 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734921 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734923 5899 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734927 5899 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734931 5899 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734934 5899 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734937 5899 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734940 5899 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734943 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734947 5899 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:24:59.740567 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734949 5899 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734952 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734955 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734958 5899 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734960 5899 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734963 5899 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734965 5899 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734968 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734970 5899 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734974 5899 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734977 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734980 5899 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734983 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734986 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734988 5899 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734991 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734994 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734996 5899 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.734999 5899 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735001 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:24:59.741028 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735004 5899 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735007 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735010 5899 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735012 5899 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735015 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735114 5899 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735130 5899 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735139 5899 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735144 5899 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735148 5899 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735151 5899 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735170 5899 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735176 5899 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735179 5899 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735182 5899 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735186 5899 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735190 5899 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735193 5899 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735196 5899 flags.go:64] FLAG: --cgroup-root="" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735199 5899 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735202 5899 flags.go:64] FLAG: --client-ca-file="" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735207 5899 flags.go:64] FLAG: --cloud-provider="external" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735210 5899 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 07:24:59.741540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735214 5899 flags.go:64] FLAG: --cluster-domain="" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735218 5899 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735221 5899 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735224 5899 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735227 5899 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735231 5899 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735234 5899 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735237 5899 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735241 5899 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735244 5899 flags.go:64] FLAG: --contention-profiling="false" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735247 5899 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735250 5899 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735253 5899 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735256 5899 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735260 5899 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735263 5899 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735266 5899 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735269 5899 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735272 5899 flags.go:64] FLAG: --enable-server="true" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735275 5899 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735281 5899 flags.go:64] FLAG: --event-burst="100" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735284 5899 flags.go:64] FLAG: --event-qps="50" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735287 5899 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735291 5899 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 07:24:59.742085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735293 5899 flags.go:64] FLAG: --eviction-hard="" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735298 5899 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735301 5899 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735305 5899 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735308 5899 flags.go:64] FLAG: --eviction-soft="" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735311 5899 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735315 5899 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735318 5899 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735321 5899 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735324 5899 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735327 5899 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735330 5899 flags.go:64] FLAG: --feature-gates="" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735334 5899 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735337 5899 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735341 5899 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735344 5899 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735347 5899 flags.go:64] FLAG: --healthz-port="10248" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735350 5899 flags.go:64] FLAG: --help="false" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735353 5899 flags.go:64] FLAG: --hostname-override="ip-10-0-135-6.ec2.internal" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735356 5899 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735359 5899 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735362 5899 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735365 5899 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 07:24:59.742709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735369 5899 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735372 5899 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735375 5899 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735378 5899 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735381 5899 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735387 5899 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735390 5899 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735393 5899 flags.go:64] FLAG: --kube-reserved="" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735396 5899 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735398 5899 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735402 5899 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735405 5899 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735408 5899 flags.go:64] FLAG: --lock-file="" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735411 5899 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735414 5899 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735419 5899 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735424 5899 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735427 5899 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735430 5899 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735433 5899 flags.go:64] FLAG: --logging-format="text" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735435 5899 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735439 5899 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735442 5899 flags.go:64] FLAG: --manifest-url="" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735445 5899 flags.go:64] FLAG: --manifest-url-header="" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735449 5899 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 07:24:59.743332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735452 5899 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735456 5899 flags.go:64] FLAG: --max-pods="110" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735459 5899 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735462 5899 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735465 5899 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735468 5899 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735471 5899 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735474 5899 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735477 5899 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735484 5899 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735487 5899 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735490 5899 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735494 5899 flags.go:64] FLAG: --pod-cidr="" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735497 5899 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735500 5899 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735503 5899 flags.go:64] FLAG: --pods-per-core="0" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735513 5899 flags.go:64] FLAG: --port="10250" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735517 5899 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735520 5899 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-097f5d70939a7d82e" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735528 5899 flags.go:64] FLAG: --qos-reserved="" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735532 5899 flags.go:64] FLAG: --read-only-port="10255" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735535 5899 flags.go:64] FLAG: --register-node="true" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735539 5899 flags.go:64] FLAG: --register-with-taints="" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735543 5899 flags.go:64] FLAG: --registry-burst="10" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735546 5899 flags.go:64] FLAG: --registry-qps="5" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735549 5899 flags.go:64] FLAG: --reserved-cpus="" Nov 02 07:24:59.743944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735552 5899 flags.go:64] FLAG: --reserved-memory="" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735556 5899 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735559 5899 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735562 5899 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735565 5899 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735568 5899 flags.go:64] FLAG: --runonce="false" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735571 5899 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735574 5899 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735576 5899 flags.go:64] FLAG: --seccomp-default="false" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735579 5899 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735582 5899 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735585 5899 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735588 5899 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735591 5899 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735594 5899 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735597 5899 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735600 5899 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735603 5899 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735607 5899 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735610 5899 flags.go:64] FLAG: --system-cgroups="" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735613 5899 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735618 5899 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735621 5899 flags.go:64] FLAG: --tls-cert-file="" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735623 5899 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735627 5899 flags.go:64] FLAG: --tls-min-version="" Nov 02 07:24:59.744602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735630 5899 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735638 5899 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735642 5899 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735646 5899 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735649 5899 flags.go:64] FLAG: --v="2" Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735657 5899 flags.go:64] FLAG: --version="false" Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735661 5899 flags.go:64] FLAG: --vmodule="" Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735665 5899 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.735668 5899 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735839 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735843 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735846 5899 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735849 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735852 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735855 5899 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735857 5899 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735860 5899 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735863 5899 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735865 5899 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735868 5899 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735871 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:24:59.745253 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735874 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735876 5899 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735879 5899 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735882 5899 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735887 5899 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735889 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735892 5899 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735894 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735897 5899 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735900 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735902 5899 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735905 5899 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735908 5899 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735917 5899 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735921 5899 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735924 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735926 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735929 5899 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735931 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735934 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:24:59.745823 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735936 5899 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735939 5899 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735941 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735944 5899 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735953 5899 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735956 5899 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735959 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735962 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735964 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735967 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735970 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735972 5899 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735975 5899 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735977 5899 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735981 5899 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735984 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735988 5899 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735991 5899 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:24:59.746513 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735994 5899 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735997 5899 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.735999 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736002 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736004 5899 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736007 5899 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736010 5899 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736012 5899 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736022 5899 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736025 5899 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736027 5899 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736030 5899 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736032 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736035 5899 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736038 5899 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736040 5899 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736043 5899 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736046 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736049 5899 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:24:59.747011 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736053 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736056 5899 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736059 5899 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736062 5899 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736064 5899 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736067 5899 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736069 5899 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736072 5899 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736075 5899 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736077 5899 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736080 5899 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736082 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736085 5899 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736088 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736090 5899 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736093 5899 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736095 5899 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736098 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736100 5899 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736103 5899 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:24:59.747561 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736105 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736109 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736117 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736120 5899 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736123 5899 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736125 5899 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736128 5899 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736130 5899 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736133 5899 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736135 5899 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736138 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736141 5899 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736143 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736146 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736149 5899 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736151 5899 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:24:59.748039 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736154 5899 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.736196 5899 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736244 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736248 5899 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736252 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736254 5899 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736257 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736259 5899 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736262 5899 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736265 5899 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736267 5899 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736270 5899 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736273 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736275 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736278 5899 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736280 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:24:59.748555 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736283 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736285 5899 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736289 5899 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736297 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736300 5899 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736303 5899 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736306 5899 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736308 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736311 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736313 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736316 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736318 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736321 5899 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736323 5899 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736326 5899 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736330 5899 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736334 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736337 5899 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:24:59.749006 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736340 5899 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736343 5899 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736346 5899 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736349 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736351 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736354 5899 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736357 5899 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736360 5899 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736362 5899 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736364 5899 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736367 5899 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736370 5899 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736372 5899 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736374 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736377 5899 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736379 5899 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736383 5899 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736386 5899 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736394 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:24:59.749626 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736398 5899 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736402 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736405 5899 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736408 5899 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736410 5899 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736413 5899 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736416 5899 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736418 5899 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736421 5899 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736423 5899 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736426 5899 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736429 5899 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736431 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736434 5899 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736436 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736439 5899 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736441 5899 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736444 5899 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736448 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736451 5899 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:24:59.750341 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736453 5899 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736456 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736458 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736461 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736463 5899 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736465 5899 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736468 5899 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736471 5899 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736473 5899 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736477 5899 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736480 5899 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736483 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736492 5899 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736495 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736498 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736500 5899 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736503 5899 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736505 5899 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736508 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736510 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:24:59.750837 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736513 5899 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736516 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736518 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736521 5899 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736524 5899 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736526 5899 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736529 5899 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736531 5899 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736534 5899 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736536 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736541 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736543 5899 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736546 5899 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736548 5899 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.736552 5899 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.741606 5899 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.741621 5899 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.741645 5899 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.741650 5899 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 07:24:59.751456 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741708 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741713 5899 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741716 5899 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741719 5899 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741722 5899 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741724 5899 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741727 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741730 5899 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741733 5899 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741735 5899 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741738 5899 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741741 5899 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741743 5899 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741746 5899 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741749 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741751 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741754 5899 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741756 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741759 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741762 5899 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:24:59.751920 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741765 5899 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741768 5899 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741771 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741774 5899 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741777 5899 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741780 5899 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741783 5899 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741786 5899 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741793 5899 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741796 5899 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741799 5899 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741801 5899 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741804 5899 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741807 5899 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741810 5899 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741814 5899 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741818 5899 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741822 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741825 5899 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:24:59.752413 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741828 5899 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741833 5899 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741837 5899 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741840 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741843 5899 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741847 5899 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741850 5899 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741858 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741862 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741864 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741867 5899 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741870 5899 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741872 5899 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741875 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741877 5899 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741880 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741882 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741885 5899 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741888 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:24:59.752953 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741891 5899 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741893 5899 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741896 5899 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741898 5899 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741901 5899 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741904 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741907 5899 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741909 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741912 5899 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741915 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741917 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741920 5899 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741922 5899 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741925 5899 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741928 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741930 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741933 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741935 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741938 5899 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741941 5899 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:24:59.753771 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741943 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741952 5899 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741955 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741957 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741960 5899 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741962 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741965 5899 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741968 5899 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741970 5899 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741973 5899 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741975 5899 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741978 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741980 5899 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741983 5899 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741985 5899 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741988 5899 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741990 5899 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741993 5899 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741995 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.741998 5899 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:24:59.754591 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742000 5899 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742003 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742005 5899 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742008 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742010 5899 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742013 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742015 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742017 5899 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.742045 5899 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742202 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742209 5899 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742212 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742215 5899 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742218 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742228 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742232 5899 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:24:59.755367 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742235 5899 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742237 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742241 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742243 5899 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742246 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742248 5899 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742251 5899 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742254 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742257 5899 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742259 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742262 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742264 5899 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742267 5899 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742269 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742272 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742274 5899 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742277 5899 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742280 5899 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742282 5899 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:24:59.755926 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742285 5899 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742287 5899 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742290 5899 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742293 5899 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742295 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742298 5899 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742300 5899 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742303 5899 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742305 5899 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742308 5899 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742311 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742313 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742321 5899 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742324 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742327 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742329 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742332 5899 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742334 5899 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742337 5899 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742339 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:24:59.756576 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742342 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742344 5899 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742347 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742349 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742352 5899 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742356 5899 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742360 5899 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742363 5899 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742366 5899 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742369 5899 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742372 5899 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742374 5899 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742377 5899 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742380 5899 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742382 5899 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742385 5899 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742387 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742391 5899 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742394 5899 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:24:59.757377 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742397 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742400 5899 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742403 5899 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742406 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742409 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742411 5899 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742417 5899 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742420 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742423 5899 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742426 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742428 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742431 5899 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742433 5899 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742436 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742438 5899 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742441 5899 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742444 5899 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742446 5899 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742449 5899 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:24:59.757838 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742451 5899 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742454 5899 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742456 5899 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742459 5899 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742461 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742464 5899 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742466 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742468 5899 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742471 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742473 5899 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742476 5899 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742479 5899 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742482 5899 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742486 5899 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742489 5899 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742491 5899 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742494 5899 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742497 5899 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742499 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742504 5899 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:24:59.758319 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742507 5899 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742509 5899 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.742533 5899 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742567 5899 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742571 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742574 5899 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742577 5899 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742580 5899 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742583 5899 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742585 5899 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742588 5899 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742590 5899 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742593 5899 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742596 5899 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742598 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742601 5899 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 07:24:59.758866 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742603 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742606 5899 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742609 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742611 5899 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742614 5899 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742617 5899 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742619 5899 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742622 5899 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742624 5899 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742627 5899 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742633 5899 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742636 5899 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742638 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742641 5899 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742643 5899 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742647 5899 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742650 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742653 5899 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742655 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742658 5899 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 07:24:59.759273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742661 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742664 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742675 5899 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742678 5899 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742681 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742684 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742686 5899 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742689 5899 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742692 5899 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742695 5899 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742697 5899 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742700 5899 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742703 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742705 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742708 5899 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742711 5899 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742713 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742716 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742719 5899 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742721 5899 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 07:24:59.759814 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742724 5899 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742727 5899 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742731 5899 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742734 5899 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742736 5899 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742739 5899 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742741 5899 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742747 5899 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742750 5899 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742752 5899 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742756 5899 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742758 5899 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742761 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742763 5899 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742766 5899 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742769 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742771 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742774 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742776 5899 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742779 5899 feature_gate.go:386] unrecognized feature gate: Example Nov 02 07:24:59.760372 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742782 5899 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742784 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742787 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742789 5899 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742792 5899 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742794 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742797 5899 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742801 5899 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742804 5899 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742807 5899 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742811 5899 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742813 5899 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742816 5899 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742818 5899 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742822 5899 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742825 5899 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742828 5899 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742830 5899 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742833 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 07:24:59.760891 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742838 5899 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742841 5899 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742844 5899 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742847 5899 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742851 5899 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742855 5899 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742857 5899 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742860 5899 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742863 5899 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742865 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742868 5899 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742870 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742873 5899 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:24:59.742875 5899 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.742926 5899 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.743906 5899 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.743963 5899 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.744140 5899 server.go:1014] "Starting client certificate rotation" Nov 02 07:24:59.761370 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.744276 5899 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 07:24:59.761803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.744325 5899 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 07:18:48 +0000 UTC" deadline="2028-04-06 07:36:34.884609271 +0000 UTC" Nov 02 07:24:59.761803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.744353 5899 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12504h11m35.140257904s" Nov 02 07:24:59.761803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.744594 5899 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 07:24:59.761803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.745615 5899 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 07:24:59.761803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.746897 5899 log.go:25] "Validated CRI v1 runtime API" Nov 02 07:24:59.761803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.748833 5899 log.go:25] "Validated CRI v1 image API" Nov 02 07:24:59.761803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.749652 5899 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 07:24:59.761803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.752232 5899 fs.go:135] Filesystem UUIDs: map[53eee952-6703-41c0-a437-6a506ab1bf7a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f3a16f42-222b-4922-bac7-4ae558733e5c:/dev/nvme0n1p4] Nov 02 07:24:59.762059 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.752252 5899 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12f8cde28adab95df4cf9180acfce8036a380819d2b0aeaf5da4fad5aa6f92a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12f8cde28adab95df4cf9180acfce8036a380819d2b0aeaf5da4fad5aa6f92a2/userdata/shm major:0 minor:275 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a2d3fbe31932c9965beea105f1435af6f4ccf22e75ffd6c5199e42e257851ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a2d3fbe31932c9965beea105f1435af6f4ccf22e75ffd6c5199e42e257851ef/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d2977f2c8aa913fba14aa8972796cc2c5a64f43bfaa7f7b2e0393c233d98fd2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d2977f2c8aa913fba14aa8972796cc2c5a64f43bfaa7f7b2e0393c233d98fd2/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e47d4a08d3395d94cd3cc3f03529bb79e42c8df4074f3ea97ae2f4c5f130336/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e47d4a08d3395d94cd3cc3f03529bb79e42c8df4074f3ea97ae2f4c5f130336/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/332528e1ae9abba2f8985c8bdde36e38f596b5208f97d9ed3292612c129390b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/332528e1ae9abba2f8985c8bdde36e38f596b5208f97d9ed3292612c129390b6/userdata/shm major:0 minor:374 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3547e0dafded1783e257bc3b835241e079ca4b199bbd1167bf4d7dc2939701e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3547e0dafded1783e257bc3b835241e079ca4b199bbd1167bf4d7dc2939701e6/userdata/shm major:0 minor:493 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a2d1c0c22f3fd36ac5cc56b309fac4676b439e476c00aa38ae5c6236cc89eb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a2d1c0c22f3fd36ac5cc56b309fac4676b439e476c00aa38ae5c6236cc89eb1/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b2ac46dfbc30ba5a825a838554b0c6d6a8a73dc25ec2ddfebe032b9eb260c97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b2ac46dfbc30ba5a825a838554b0c6d6a8a73dc25ec2ddfebe032b9eb260c97/userdata/shm major:0 minor:489 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/545680a331c78bec9728827fd32a150b095c3cc57a23b2dc24cc15a2ec90a9be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/545680a331c78bec9728827fd32a150b095c3cc57a23b2dc24cc15a2ec90a9be/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59e951ef66835dbeebaa695e657955359cb030eff48402f0d5dbe9463ba0be46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59e951ef66835dbeebaa695e657955359cb030eff48402f0d5dbe9463ba0be46/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/652ddf883f834360985f8af6947f12b2139cdbdff0c5cf725fcd714a869d2d2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/652ddf883f834360985f8af6947f12b2139cdbdff0c5cf725fcd714a869d2d2a/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78d4343c996b21ad40631bc0d272b81a513f5afc39e60b05292139db33c19d15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78d4343c996b21ad40631bc0d272b81a513f5afc39e60b05292139db33c19d15/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/792a2c51328c1027a6c630f5c72143d25ecd4946240c1adf01746da581a9a899/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/792a2c51328c1027a6c630f5c72143d25ecd4946240c1adf01746da581a9a899/userdata/shm major:0 minor:435 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dbf748226a4d9ef4d3fcb6b13a8eb00667daf6284bc74941652929074bc9dcb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dbf748226a4d9ef4d3fcb6b13a8eb00667daf6284bc74941652929074bc9dcb/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82938c87d260f19af2189bdcbedcd4f36a06bb6d0d4719ea1975bc288ba4ae72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82938c87d260f19af2189bdcbedcd4f36a06bb6d0d4719ea1975bc288ba4ae72/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/846c1b684fdfc2efe50a4364b2096eeee11c6f4832421b4f6784cbed929be4b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/846c1b684fdfc2efe50a4364b2096eeee11c6f4832421b4f6784cbed929be4b0/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/855497f8c5e330e05adc47031d251793180839b97bcdd7d53a2af0bc1a6b53c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/855497f8c5e330e05adc47031d251793180839b97bcdd7d53a2af0bc1a6b53c8/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8df284fb36e6a77a897fa231542dcdbde2bcce714111338c842b39b67a0e4a72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8df284fb36e6a77a897fa231542dcdbde2bcce714111338c842b39b67a0e4a72/userdata/shm major:0 minor:491 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92d751fe6bb0ba81d295c92ae8ca43233805fd440339cad982dd6fc8c293ad43/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92d751fe6bb0ba81d295c92ae8ca43233805fd440339cad982dd6fc8c293ad43/userdata/shm major:0 minor:439 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/950f78d0d4d456f3acc78856c9ea792bdbe905ec2f628ed1070463c03a3bbc90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/950f78d0d4d456f3acc78856c9ea792bdbe905ec2f628ed1070463c03a3bbc90/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9cb3cd7090bb43620924b05554a98a225161bf986b525b7c7da5992d7b25a593/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9cb3cd7090bb43620924b05554a98a225161bf986b525b7c7da5992d7b25a593/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b41ad2171c48e209b8c13466d305258cf3b1051788f44243212abdadebfb1836/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b41ad2171c48e209b8c13466d305258cf3b1051788f44243212abdadebfb1836/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1bdf19a73e7982a74f1c4ffc53fc327d44336729db358619c1a892ec8604367/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1bdf19a73e7982a74f1c4ffc53fc327d44336729db358619c1a892ec8604367/userdata/shm major:0 minor:321 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5b94dbfaf37d5068efe781f58c18c6f943469de2799ebe465ad7e2411c93af4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5b94dbfaf37d5068efe781f58c18c6f943469de2799ebe465ad7e2411c93af4/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3158881f97bbf37e6370694d1d8e74353afd34e0a3a428d917ce8707f7cd0f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3158881f97bbf37e6370694d1d8e74353afd34e0a3a428d917ce8707f7cd0f8/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0768e3b2-d790-4375-acf5-111424b1c5c3/volumes/kubernetes.io~projected/kube-api-access-xrwkv:{mountpoint:/var/lib/kubelet/pods/0768e3b2-d790-4375-acf5-111424b1c5c3/volumes/kubernetes.io~projected/kube-api-access-xrwkv major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0768e3b2-d790-4375-acf5-111424b1c5c3/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0768e3b2-d790-4375-acf5-111424b1c5c3/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0768e3b2-d790-4375-acf5-111424b1c5c3/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0768e3b2-d790-4375-acf5-111424b1c5c3/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c592b3d-c373-48f3-a1b4-e75b8134afd6/volumes/kubernetes.io~projected/kube-api-access-qjrb9:{mountpoint:/var/lib/kubelet/pods/0c592b3d-c373-48f3-a1b4-e75b8134afd6/volumes/kubernetes.io~projected/kube-api-access-qjrb9 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c592b3d-c373-48f3-a1b4-e75b8134afd6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/0c592b3d-c373-48f3-a1b4-e75b8134afd6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e577f03-1ea2-4496-9adb-84a2dc4dc72f/volumes/kubernetes.io~projected/kube-api-access-5hrz7:{mountpoint:/var/lib/kubelet/pods/0e577f03-1ea2-4496-9adb-84a2dc4dc72f/volumes/kubernetes.io~projected/kube-api-access-5hrz7 major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e577f03-1ea2-4496-9adb-84a2dc4dc72f/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/0e577f03-1ea2-4496-9adb-84a2dc4dc72f/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e577f03-1ea2-4496-9adb-84a2dc4dc72f/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/0e577f03-1ea2-4496-9adb-84a2dc4dc72f/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e577f03-1ea2-4496-9adb-84a2dc4dc72f/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/0e577f03-1ea2-4496-9adb-84a2dc4dc72f/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b808d2a-56c8-477f-bca0-d9d69d711516/volumes/kubernetes.io~projected/kube-api-access-d6mt7:{mountpoint:/var/lib/kubelet/pods/1b808d2a-56c8-477f-bca0-d9d69d711516/volumes/kubernetes.io~projected/kube-api-access-d6mt7 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b808d2a-56c8-477f-bca0-d9d69d711516/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1b808d2a-56c8-477f-bca0-d9d69d711516/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/253a4509-48e5-48c2-8c13-2dc361e5763f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/253a4509-48e5-48c2-8c13-2dc361e5763f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:483 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~projected/kube-api-access-2mxbc:{mountpoint:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~projected/kube-api-access-2mxbc major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2eeba619-ec6d-4ec8-828b-2f409d53630d/volumes/kubernetes.io~projected/kube-api-access-659fv:{mountpoint:/var/lib/kubelet/pods/2eeba619-ec6d-4ec8-828b-2f409d53630d/volumes/kubernetes.io~projected/kube-api-access-659fv major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3201efd6-0d33-4323-bab0-b76338d090ed/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3201efd6-0d33-4323-bab0-b76338d090ed/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3201efd6-0d33-4323-bab0-b76338d090ed/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3201efd6-0d33-4323-bab0-b76338d090ed/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3201efd6-0d33-4323-bab0-b76338d090ed/volumes/kubernetes.io~projected/kube-api-access-c7m65:{mountpoint:/var/lib/kubelet/pods/3201efd6-0d33-4323-bab0-b76338d090ed/volumes/kubernetes.io~projected/kube-api-access-c7m65 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volume-subpaths/web-config/alertmanager/9 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volume-subpaths/web-config/config-reloader/6 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~empty-dir/config-out major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~projected/kube-api-access-h6mbb:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~projected/kube-api-access-h6mbb major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~projected/tls-assets major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/config-volume major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/web-config major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volume-subpaths/web-config/config-reloader/2 major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volume-subpaths/web-config/prometheus/15 major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~empty-dir/config-out major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~projected/kube-api-access-n4hv6:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~projected/kube-api-access-n4hv6 major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~projected/tls-assets major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/config major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/web-config major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71998a34-5575-4ae0-8892-2c9e88b1b396/volumes/kubernetes.io~projected/kube-api-access-pj2mr:{mountpoint:/var/lib/kubelet/pods/71998a34-5575-4ae0-8892-2c9e88b1b396/volumes/kubernetes.io~projected/kube-api-access-pj2mr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71998a34-5575-4ae0-8892-2c9e88b1b396/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/71998a34-5575-4ae0-8892-2c9e88b1b396/volumes/kubernetes.io~secret/metrics-certs major:0 minor:487 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77af1f03-6bc5-4dad-8646-eae4f5ae40f0/volumes/kubernetes.io~projected/kube-api-access-cg2fk:{mountpoint:/var/lib/kubelet/pods/77af1f03-6bc5-4dad-8646-eae4f5ae40f0/volumes/kubernetes.io~projected/kube-api-access-cg2fk major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85e9ffec-236a-4a8c-8dd1-7014e032e180/volumes/kubernetes.io~projected/kube-api-access-lvtjs:{mountpoint:/var/lib/kubelet/pods/85e9ffec-236a-4a8c-8dd1-7014e032e180/volumes/kubernetes.io~projected/kube-api-access-lvtjs major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7/volumes/kubernetes.io~projected/kube-api-access-l7xw9:{mountpoint:/var/lib/kubelet/pods/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7/volumes/kubernetes.io~projected/kube-api-access-l7xw9 major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759/volumes/kubernetes.io~projected/kube-api-access-8fk87:{mountpoint:/var/lib/kubelet/pods/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759/volumes/kubernetes.io~projected/kube-api-access-8fk87 major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/973a9b8f-405e-4dc4-8322-4d12a152178d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/973a9b8f-405e-4dc4-8322-4d12a152178d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/973a9b8f-405e-4dc4-8322-4d12a152178d/volumes/kubernetes.io~projected/kube-api-access-p6lln:{mountpoint:/var/lib/kubelet/pods/973a9b8f-405e-4dc4-8322-4d12a152178d/volumes/kubernetes.io~projected/kube-api-access-p6lln major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/973a9b8f-405e-4dc4-8322-4d12a152178d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/973a9b8f-405e-4dc4-8322-4d12a152178d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1349fb3-19b0-4648-92e1-fcd2da2076bf/volumes/kubernetes.io~projected/kube-api-access-5wkph:{mountpoint:/var/lib/kubelet/pods/a1349fb3-19b0-4648-92e1-fcd2da2076bf/volumes/kubernetes.io~projected/kube-api-access-5wkph major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02/volumes/kubernetes.io~projected/kube-api-access-dbk9t:{mountpoint:/var/lib/kubelet/pods/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02/volumes/kubernetes.io~projected/kube-api-access-dbk9t major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0/volumes/kubernetes.io~projected/kube-api-access-d6nkn:{mountpoint:/var/lib/kubelet/pods/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0/volumes/kubernetes.io~projected/kube-api-access-d6nkn major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6e945a4-9c6d-4310-902f-eff8fc826621/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d6e945a4-9c6d-4310-902f-eff8fc826621/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e229a07c-7415-4af9-a9eb-9e7984e0a510/volumes/kubernetes.io~projected/kube-api-access-2xftv:{mountpoint:/var/lib/kubelet/pods/e229a07c-7415-4af9-a9eb-9e7984e0a510/volumes/kubernetes.io~projected/kube-api-access-2xftv major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e76535e7-a981-4b8a-9bcf-5e400d35c1d1/volumes/kubernetes.io~projected/kube-api-access-4zx4h:{mountpoint:/var/lib/kubelet/pods/e76535e7-a981-4b8a-9bcf-5e400d35c1d1/volumes/kubernetes.io~projected/kube-api-access-4zx4h major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e76535e7-a981-4b8a-9bcf-5e400d35c1d1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e76535e7-a981-4b8a-9bcf-5e400d35c1d1/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef891d1c-0c5c-45b3-98eb-563324d8822f/volumes/kubernetes.io~projected/kube-api-access-w8qfz:{mountpoint:/var/lib/kubelet/pods/ef891d1c-0c5c-45b3-98eb-563324d8822f/volumes/kubernetes.io~projected/kube-api-access-w8qfz major:0 minor:488 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdb3f2d5-5e21-48c3-af74-eb8fd530046a/volumes/kubernetes.io~projected/kube-api-access-djnzm:{mountpoint:/var/lib/kubelet/pods/fdb3f2d5-5e21-48c3-af74-eb8fd530046a/volumes/kubernetes.io~projected/kube-api-access-djnzm major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ab39c5746bd4f8cfc5d5cdb1f71d530f379784b9235c7fd91b498c7129bea93f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8be5d78b6978e1024a20d7a7ca15b58078a92b87c99229e8668fb690a60ac4b2/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/26c39f6d25c4cac1186aefbacd81d6e9f75bd3b932cfe93c3a482f7b7efeab80/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d3cbd3aabce38bad35969857cf781ab0c77e188c1136ccc33e82eb44da2157d6/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/150b0414f76d5df350602d519eccc402f689a635bda05ae2e615bcb0681b2cab/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c98829d414fd4e3868cb8951d9b44e88900b65e0277e8c5663224fd8f3689a6d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/28f675a51a6ea0ec906426b8248d20fb6cb16642907359482e3e8fb988645632/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b753cb4245bf5dd6b8a4583200e93a1025f26440d9d548a076b1aa7760d9954f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/6ddc846f869175420bacb44a48e9c76986291deae5c62a908aa0bf813bd6f92b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/4d57a7e8407916eef95efca78b758ed4509d3d3fb027bde5b46e1f25fc3ec69b/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/3221d1bc66d33300798ad6f132cc219ef6420d726d1ca2c3e33f083b8a2ebf32/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/ae7273feea45f649464e8ab871d6688ae80cb1b339e15bee7f4a374b584e85bb/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/717f72ac12051a6bb4a4b7eea26a65bfd3bd48c503e07d34645abc9322dc71e3/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/f90f40df772f6f28dc43737b89504f5f3ccd75a9a2de3bcadc0ca0f0fa5197d0/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/cfa5cf8971adba007f25ae4eef5f05880da27291da6e36d4e1ff5999956964b1/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d37e1e3379f303af8a384480205e287985067d00d79d1e9a158653b97d3690f0/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c1fab220cb3377f6842bdf1183f5e1f497255bc57c76c6ebd21cd35390e76218/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c460705ab80dd633a33550fa981d4bc919921a4b890ef23a1a7e41f213076672/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/dc22426bd3ddfbd029cfc0ab378a9b6d7668c82f325f1cb7d9e9b5e83af82902/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d5b5dbf7f94636509b2b86aa9d1b4b32557e9fb2ac5b5de6e219ad49494babef/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/984c75c60978755753b6d3f314173989372e5ab2c3b22e20d73835f5dca26b0c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/463f03849e43bf3ff7fca99e043126ad4af961c65fdee0dce6b030099f122c44/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/db0d9385f815213c7de98b083efe35d4da05ec584a8789a1c673ca5f4645749a/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/f18cbbc5aa542fcf108c9b68a172d09de8397d59219bbc41aa9a3de4e39888dc/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/071788189e97a04e0077118ff3143ae0e03ea589e732cdea386f926b63e78104/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/2682d444a9b9017b81136239e005dbcd2fcebaaf6f6724ac226a44f2cd776d55/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/fff3d93f9e01a025a3f37b95caca5d0f6298a3ad0fb0ea09119f17442a9b04a6/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/8357268e7925beb5c8b2e46eca17d6c771e377ca3fa147c485ef2860c260a82b/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/8eb3e29383e2a79c96c4fc6c64325e9ee68e3eea806b6c9b66af80cc3609be2c/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/67e370b98b5be5bdbd8c78ba8a990ae3b6dea71e1845a9c04cc075936919f1c7/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/d1f3dc24a3c9953ad5ed96f1c3be16c4ae817c255c945303d8a4e4baf6f5fbb3/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/fd38da06ca89a6f5156a644fe0ea07976286bc52f14fd0728e7b1848565b43bb/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/95e81ebd1bd5b32e6eb1652690288589245dde2828021344965ba6a80bb64007/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/12d1713f63a52afb1479f194e638de4d8b7004247264e9529baed2ec38f82f07/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/8ddf6346a8f621981def46a76d6fe107803018118bf6bd52b18592136da773bf/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/2d7293b5a943585b160a7cfb88a03522b314543751c61fc53c0fad5be04e7ca0/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/896cae0a2776ec07026260f74aada51ce5572ed2fd991e95e99e95535153c1c0/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/f6d253a024d20f0fafadfa5d4e396ddb9e1fb54da3045c563d1a065db8eaf80d/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/597413ab6a4b589f5bc33fc99a3b5342550061e635f667b815e33857169c105b/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/46d637c1b48d48cb2a4e2cb9f123bc5ba73f2b385ba7c4d53db674141977bdf2/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/e63ab05fffa9457241948eeb65a2f60d9b7bf80517a81008e16af5db98022337/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/9885b11d28cc6e8b243eb7b60545b1ccac80c755cd586e4ee65330053118b736/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/de9b8a3996662f67079bdafb8058280fc23603b297bf5a99326aaf7c5358d266/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/3cb2028c85e637f4a5f8baa7e23717d47f446fdbdfc4dcfe92db78a54797b253/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/9e15dcbf58b568c06b6a9b339da58685b68995c91ef3196cee8615e690137f65/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/4424d43c215ec4bb564fad0d540c6822ee38550dae3c144410165514dc9c42f2/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/693c705cc2a87d533f1585b530e1953ed1b92745e9343b341916a6e195772a9d/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/dda986709d9a871768eec5a2032903c4dfda87e5916c719c33f5f45066ced0ae/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/fb355c1b70edb550b0a927579df8a624fdeb6563cd8f0bff7aff95f6f548af57/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/698f9c7ce1cf5372057e1bdc3f77f1ff24136b06496e0bc8d01ee938e2c048b8/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/47ee6e35feeda854ed5b68d72ae977a7d3316bc310c683546a47645c45244a40/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/d6a6be23764a7e7806f6a67caf3f2e56b9b5fce88d198ab7f5cb4ba45edaa289/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/f63315186b4b8e9d7b68ced6642a94ad50e501648249dba70889a42840006dad/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/2d744ba616db122a06046e1d4d4d3cbe07c0450b5b8f77c21c1351276d1aab03/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/1e149b78a00e0d3f4f6c8d41616ae12661caddd250027929c6b2c3c717cd58df/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/8c15e2e24dda7bf0905ce61368866b35dcc9560ab0c58fc33c892480209236bd/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/ce7cfe70441583d6d7e45f5b7a557b5d6ec5ee0f67634e64c72602a3d5c8f95e/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/df1f81386a27f162b39a87879cd6fe3a606a78b5fe39627431bd70919ec7c021/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/02c576b4c068ca3f7db033167e245e221bba9704a8e32005a8a483bc62c760fb/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/92f4bb0109272147b5083d5a6afc1799c6db080895e5b36b110fe58ac6f86c7b/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/1fd94dc399181a91a8233cb0c50ab92de4691ef029352d91878b13b59481bae4/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/68abd99ca71df373984e68e5ce1b4166b47fa139661ce3c2d9d8f4299ea567b2/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/a697fb1645a7b1c4ecf938ffdd32bb1054490033fb644c1dd0b4897e2fdb7e4e/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/30b12ac685f37616830628a670698cf8a2c5cec2255b34a58de1e3a26a96a760/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-463:{mountpoint:/var/lib/containers/storage/overlay/6e7fbfdd9d0521fc7be6fed7d641a3901952d428d1088aba22e8401943792a08/merged major:0 minor:463 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/a1a5884563994ebe98dad09e9386f8c3f4bd98f44445cf284ae440cf6c4cb0b6/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/65500c2b3a7d240cbbf9b3417f291bcc8920a4709142e9b5e4711b5dc075765a/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/52bc124c0a2d65d4f1efab0490fec93a151bd3233aa2f336fcc34a0356c1f357/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/9b39940744056c22b592c9358003fc52ae33334f37b6540f9b7d5d533e2baad3/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/c80e7ee9f8a51dd3a715d156061e24396b5e31c764d3f7704742fdfa36dfa145/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-497:{mountpoint:/var/lib/containers/storage/overlay/925fcc52242da81c61ece75fc622fb322376d6f080c25b4a0f4783cfb0f08a57/merged major:0 minor:497 fsType:overlay blockSize:0} overlay_0-499:{mountpoint:/var/lib/containers/storage/overlay/ca151986f5c1b2e01f918602d02bfaba80122a6cac8a4b185e54305c24ea37bb/merged major:0 minor:499 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/bce31d734b635ddb01914d62a3b240719217fbd00d423900adff1c8b1a8b8066/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/cb49f4368a66316a4d50a8de3c7d2a6edf3a6f6a0546a68af056fed636349be3/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-509:{mountpoint:/var/lib/containers/storage/overlay/cad6bb24fb57d38bfe9781ce5ee02091d8946a789c47743bfc6f82ca2bdae1a3/merged major:0 minor:509 fsType:overlay blockSize:0} overlay_0-514:{mountpoint:/var/lib/containers/storage/overlay/10abef210450dd0d6ffe7739bad40ac56666e15db8504b0ffb688cff0d19b304/merged major:0 minor:514 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/54cac9bb58524136eecd2e2e9b8ab4c2b3823e713884ba05f181eac734d17f09/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/0b4d9c553f1781355515c5fa65a511831d2fbbb03a2a6f17f5758a7fc688518d/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/71e149d7e5ee1f5e06121b2d9ab7a8ab35fb36aae772bc2d9a6d3548d000a5be/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/483503c6413c52ddbc37990897c9b74bdb23157d8efc4ef361bf8ebb66202697/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/7bbb2d323763cf038bcd01c9c6804ef750c954df4bfa2852e47f33101a80d571/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/a27f8d40e917660b2206d89c9d4d3f2980d25e52debd67664aab1e04e59a7035/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/3c7954c5cca60592143c175548f63d137f6d87515128211a2ed7069f175cd11c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c61b899cd154065af481d8510079b689984b0f50163bd41fd53828c496ff33b5/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/08f1523e11f5c9dcd8bb73537a2b1c2a5db65a5743dfdc97ba97ef869f5a7621/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 07:24:59.762524 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.761559 5899 manager.go:217] Machine: {Timestamp:2026-11-02 07:24:59.76108668 +0000 UTC m=+0.059367498 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100896 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21640333e3786bc122adcd83e39fca SystemUUID:ec216403-33e3-786b-c122-adcd83e39fca BootID:857520d6-fcfc-4003-aef2-78b6f591d457 Filesystems:[{Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:431 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8df284fb36e6a77a897fa231542dcdbde2bcce714111338c842b39b67a0e4a72/userdata/shm DeviceMajor:0 DeviceMinor:491 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/950f78d0d4d456f3acc78856c9ea792bdbe905ec2f628ed1070463c03a3bbc90/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759/volumes/kubernetes.io~projected/kube-api-access-8fk87 DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-463 DeviceMajor:0 DeviceMinor:463 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0768e3b2-d790-4375-acf5-111424b1c5c3/volumes/kubernetes.io~projected/kube-api-access-xrwkv DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/792a2c51328c1027a6c630f5c72143d25ecd4946240c1adf01746da581a9a899/userdata/shm DeviceMajor:0 DeviceMinor:435 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:344 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2eeba619-ec6d-4ec8-828b-2f409d53630d/volumes/kubernetes.io~projected/kube-api-access-659fv DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~projected/kube-api-access-h6mbb DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/332528e1ae9abba2f8985c8bdde36e38f596b5208f97d9ed3292612c129390b6/userdata/shm DeviceMajor:0 DeviceMinor:374 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92d751fe6bb0ba81d295c92ae8ca43233805fd440339cad982dd6fc8c293ad43/userdata/shm DeviceMajor:0 DeviceMinor:439 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5b94dbfaf37d5068efe781f58c18c6f943469de2799ebe465ad7e2411c93af4/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e47d4a08d3395d94cd3cc3f03529bb79e42c8df4074f3ea97ae2f4c5f130336/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:416 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3547e0dafded1783e257bc3b835241e079ca4b199bbd1167bf4d7dc2939701e6/userdata/shm DeviceMajor:0 DeviceMinor:493 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e76535e7-a981-4b8a-9bcf-5e400d35c1d1/volumes/kubernetes.io~projected/kube-api-access-4zx4h DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:424 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:425 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:425 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6e945a4-9c6d-4310-902f-eff8fc826621/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/85e9ffec-236a-4a8c-8dd1-7014e032e180/volumes/kubernetes.io~projected/kube-api-access-lvtjs DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a1349fb3-19b0-4648-92e1-fcd2da2076bf/volumes/kubernetes.io~projected/kube-api-access-5wkph DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12f8cde28adab95df4cf9180acfce8036a380819d2b0aeaf5da4fad5aa6f92a2/userdata/shm DeviceMajor:0 DeviceMinor:275 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:428 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-514 DeviceMajor:0 DeviceMinor:514 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78d4343c996b21ad40631bc0d272b81a513f5afc39e60b05292139db33c19d15/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3158881f97bbf37e6370694d1d8e74353afd34e0a3a428d917ce8707f7cd0f8/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:426 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9cb3cd7090bb43620924b05554a98a225161bf986b525b7c7da5992d7b25a593/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dbf748226a4d9ef4d3fcb6b13a8eb00667daf6284bc74941652929074bc9dcb/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e76535e7-a981-4b8a-9bcf-5e400d35c1d1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1bdf19a73e7982a74f1c4ffc53fc327d44336729db358619c1a892ec8604367/userdata/shm DeviceMajor:0 DeviceMinor:321 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0e577f03-1ea2-4496-9adb-84a2dc4dc72f/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:421 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3201efd6-0d33-4323-bab0-b76338d090ed/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~projected/kube-api-access-2mxbc DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:425 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/253a4509-48e5-48c2-8c13-2dc361e5763f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:483 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d2977f2c8aa913fba14aa8972796cc2c5a64f43bfaa7f7b2e0393c233d98fd2/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:311 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:432 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e229a07c-7415-4af9-a9eb-9e7984e0a510/volumes/kubernetes.io~projected/kube-api-access-2xftv DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b808d2a-56c8-477f-bca0-d9d69d711516/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7/volumes/kubernetes.io~projected/kube-api-access-l7xw9 DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3201efd6-0d33-4323-bab0-b76338d090ed/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0768e3b2-d790-4375-acf5-111424b1c5c3/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:274 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:427 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:425 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/846c1b684fdfc2efe50a4364b2096eeee11c6f4832421b4f6784cbed929be4b0/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0e577f03-1ea2-4496-9adb-84a2dc4dc72f/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59e951ef66835dbeebaa695e657955359cb030eff48402f0d5dbe9463ba0be46/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:429 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-497 DeviceMajor:0 DeviceMinor:497 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-499 DeviceMajor:0 DeviceMinor:499 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c592b3d-c373-48f3-a1b4-e75b8134afd6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/973a9b8f-405e-4dc4-8322-4d12a152178d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02/volumes/kubernetes.io~projected/kube-api-access-dbk9t DeviceMajor:0 DeviceMinor:422 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdb3f2d5-5e21-48c3-af74-eb8fd530046a/volumes/kubernetes.io~projected/kube-api-access-djnzm DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/855497f8c5e330e05adc47031d251793180839b97bcdd7d53a2af0bc1a6b53c8/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82938c87d260f19af2189bdcbedcd4f36a06bb6d0d4719ea1975bc288ba4ae72/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0e577f03-1ea2-4496-9adb-84a2dc4dc72f/volumes/kubernetes.io~projected/kube-api-access-5hrz7 DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71998a34-5575-4ae0-8892-2c9e88b1b396/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:487 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef891d1c-0c5c-45b3-98eb-563324d8822f/volumes/kubernetes.io~projected/kube-api-access-w8qfz DeviceMajor:0 DeviceMinor:488 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/973a9b8f-405e-4dc4-8322-4d12a152178d/volumes/kubernetes.io~projected/kube-api-access-p6lln DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ebdbc81-5857-429e-9a08-2143afcdc097/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:315 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:430 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~projected/kube-api-access-n4hv6 DeviceMajor:0 DeviceMinor:434 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/652ddf883f834360985f8af6947f12b2139cdbdff0c5cf725fcd714a869d2d2a/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0e577f03-1ea2-4496-9adb-84a2dc4dc72f/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:423 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:433 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/77af1f03-6bc5-4dad-8646-eae4f5ae40f0/volumes/kubernetes.io~projected/kube-api-access-cg2fk DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0768e3b2-d790-4375-acf5-111424b1c5c3/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b2ac46dfbc30ba5a825a838554b0c6d6a8a73dc25ec2ddfebe032b9eb260c97/userdata/shm DeviceMajor:0 DeviceMinor:489 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-509 DeviceMajor:0 DeviceMinor:509 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71998a34-5575-4ae0-8892-2c9e88b1b396/volumes/kubernetes.io~projected/kube-api-access-pj2mr DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a2d1c0c22f3fd36ac5cc56b309fac4676b439e476c00aa38ae5c6236cc89eb1/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b808d2a-56c8-477f-bca0-d9d69d711516/volumes/kubernetes.io~projected/kube-api-access-d6mt7 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0c592b3d-c373-48f3-a1b4-e75b8134afd6/volumes/kubernetes.io~projected/kube-api-access-qjrb9 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/545680a331c78bec9728827fd32a150b095c3cc57a23b2dc24cc15a2ec90a9be/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a2d3fbe31932c9965beea105f1435af6f4ccf22e75ffd6c5199e42e257851ef/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/973a9b8f-405e-4dc4-8322-4d12a152178d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3201efd6-0d33-4323-bab0-b76338d090ed/volumes/kubernetes.io~projected/kube-api-access-c7m65 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b41ad2171c48e209b8c13466d305258cf3b1051788f44243212abdadebfb1836/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0/volumes/kubernetes.io~projected/kube-api-access-d6nkn DeviceMajor:0 DeviceMinor:347 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:12f8cde28adab95 MacAddress:ba:08:ae:52:8b:bb Speed:10000 Mtu:8901} {Name:2d2977f2c8aa913 MacAddress:22:f6:cb:f5:3b:e7 Speed:10000 Mtu:8901} {Name:332528e1ae9abba MacAddress:c2:f3:55:f1:3a:3e Speed:10000 Mtu:8901} {Name:3547e0dafded178 MacAddress:26:9b:64:bb:29:4c Speed:10000 Mtu:8901} {Name:3b2ac46dfbc30ba MacAddress:52:ad:72:59:81:8e Speed:10000 Mtu:8901} {Name:59e951ef66835db MacAddress:06:b6:d5:14:4a:b3 Speed:10000 Mtu:8901} {Name:792a2c51328c102 MacAddress:b6:bb:d0:06:68:61 Speed:10000 Mtu:8901} {Name:846c1b684fdfc2e MacAddress:6a:f0:07:71:4b:42 Speed:10000 Mtu:8901} {Name:855497f8c5e330e MacAddress:ee:af:a1:85:a2:df Speed:10000 Mtu:8901} {Name:8df284fb36e6a77 MacAddress:ee:0b:5a:6e:35:c8 Speed:10000 Mtu:8901} {Name:92d751fe6bb0ba8 MacAddress:9e:46:54:f4:6b:2d Speed:10000 Mtu:8901} {Name:950f78d0d4d456f MacAddress:52:87:e9:f0:5d:ce Speed:10000 Mtu:8901} {Name:b41ad2171c48e20 MacAddress:3e:1d:99:20:47:5a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f2:b3:d6:e5 Speed:0 Mtu:9001} {Name:br-int MacAddress:2a:5a:3d:a9:94:39 Speed:0 Mtu:8901} {Name:c1bdf19a73e7982 MacAddress:6a:42:22:73:c8:a5 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f2:b3:d6:e5 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:c6:60:c3:0a:0f:47 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:11:f5:fc:e0:61 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 07:24:59.762524 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.762080 5899 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 07:24:59.762524 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.762131 5899 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 07:24:59.762996 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.762553 5899 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 07:24:59.762996 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.762589 5899 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-6.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 07:24:59.762996 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.762778 5899 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 07:24:59.762996 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.762791 5899 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 07:24:59.762996 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.762809 5899 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 07:24:59.762996 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.762834 5899 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 07:24:59.762996 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.762959 5899 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 07:24:59.763344 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.763224 5899 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 07:24:59.763400 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.763369 5899 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 07:24:59.763448 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.763415 5899 kubelet.go:493] "Attempting to sync node with API server" Nov 02 07:24:59.763448 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.763429 5899 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 07:24:59.763448 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.763445 5899 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 07:24:59.763569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.763459 5899 kubelet.go:405] "Adding apiserver pod source" Nov 02 07:24:59.763569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.763471 5899 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 07:24:59.764734 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.764718 5899 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 07:24:59.764734 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.764732 5899 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 07:24:59.765261 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.765232 5899 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 07:24:59.767893 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.767879 5899 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 07:24:59.767893 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.767893 5899 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 07:24:59.767989 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.767980 5899 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 07:24:59.768191 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768180 5899 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 07:24:59.768231 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768210 5899 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 07:24:59.768309 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768300 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 07:24:59.768339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768314 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 07:24:59.768339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768321 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 07:24:59.768339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768327 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 07:24:59.768339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768335 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 07:24:59.768437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768341 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 07:24:59.768437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768347 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 07:24:59.768437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768353 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 07:24:59.768437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768359 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 07:24:59.768437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768365 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 07:24:59.768437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768375 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 07:24:59.768437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768385 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 07:24:59.768437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768407 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 07:24:59.768437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.768413 5899 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 07:24:59.771717 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.771702 5899 server.go:1263] "Started kubelet" Nov 02 07:24:59.772210 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.771783 5899 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 07:24:59.772210 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.771877 5899 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 07:24:59.772210 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.772002 5899 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 07:24:59.772391 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.772343 5899 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 07:24:59.772301 ip-10-0-135-6 systemd[1]: Started Kubernetes Kubelet. Nov 02 07:24:59.776413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.776393 5899 server.go:317] "Adding debug handlers to kubelet server" Nov 02 07:24:59.779660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.779644 5899 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 07:24:59.779865 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.779671 5899 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 07:24:59.779865 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.779712 5899 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 07:18:50 +0000 UTC" deadline="2028-04-19 22:12:40.755011243 +0000 UTC" Nov 02 07:24:59.779865 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.779729 5899 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12830h47m40.97528501s" Nov 02 07:24:59.779865 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.779753 5899 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 07:24:59.779865 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.779763 5899 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 07:24:59.779865 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.779861 5899 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 07:24:59.780156 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:24:59.779876 5899 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-6.ec2.internal\" not found" Nov 02 07:24:59.781204 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.780708 5899 factory.go:55] Registering systemd factory Nov 02 07:24:59.781204 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.780732 5899 factory.go:223] Registration of the systemd container factory successfully Nov 02 07:24:59.782482 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782454 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ebdbc81-5857-429e-9a08-2143afcdc097" volumeName="kubernetes.io/projected/2ebdbc81-5857-429e-9a08-2143afcdc097-kube-api-access-2mxbc" seLinuxMountContext="" Nov 02 07:24:59.782531 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782484 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ebdbc81-5857-429e-9a08-2143afcdc097" volumeName="kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-grpc-tls" seLinuxMountContext="" Nov 02 07:24:59.782531 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782495 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067c1c4-cfc8-4358-9069-f24576389879" volumeName="kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-metrics-client-ca" seLinuxMountContext="" Nov 02 07:24:59.782531 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782503 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85e9ffec-236a-4a8c-8dd1-7014e032e180" volumeName="kubernetes.io/configmap/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-daemon-config" seLinuxMountContext="" Nov 02 07:24:59.782531 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782511 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ebdbc81-5857-429e-9a08-2143afcdc097" volumeName="kubernetes.io/empty-dir/2ebdbc81-5857-429e-9a08-2143afcdc097-tmp" seLinuxMountContext="" Nov 02 07:24:59.782531 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782519 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config" seLinuxMountContext="" Nov 02 07:24:59.782531 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782526 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85e9ffec-236a-4a8c-8dd1-7014e032e180" volumeName="kubernetes.io/projected/85e9ffec-236a-4a8c-8dd1-7014e032e180-kube-api-access-lvtjs" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782534 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ebdbc81-5857-429e-9a08-2143afcdc097" volumeName="kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782544 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782557 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7" volumeName="kubernetes.io/secret/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-openshift-state-metrics-tls" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782565 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6e945a4-9c6d-4310-902f-eff8fc826621" volumeName="kubernetes.io/secret/d6e945a4-9c6d-4310-902f-eff8fc826621-agent-certs" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782572 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e76535e7-a981-4b8a-9bcf-5e400d35c1d1" volumeName="kubernetes.io/projected/e76535e7-a981-4b8a-9bcf-5e400d35c1d1-kube-api-access-4zx4h" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782580 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e577f03-1ea2-4496-9adb-84a2dc4dc72f" volumeName="kubernetes.io/configmap/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782586 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782595 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e577f03-1ea2-4496-9adb-84a2dc4dc72f" volumeName="kubernetes.io/projected/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-kube-api-access-5hrz7" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782603 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067c1c4-cfc8-4358-9069-f24576389879" volumeName="kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-config-out" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782609 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="253a4509-48e5-48c2-8c13-2dc361e5763f" volumeName="kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782629 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782636 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdb3f2d5-5e21-48c3-af74-eb8fd530046a" volumeName="kubernetes.io/projected/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-kube-api-access-djnzm" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782643 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782650 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-grpc-tls" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782657 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="973a9b8f-405e-4dc4-8322-4d12a152178d" volumeName="kubernetes.io/secret/973a9b8f-405e-4dc4-8322-4d12a152178d-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782678 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" volumeName="kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-oauth-serving-cert" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782690 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e229a07c-7415-4af9-a9eb-9e7984e0a510" volumeName="kubernetes.io/projected/e229a07c-7415-4af9-a9eb-9e7984e0a510-kube-api-access-2xftv" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782698 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c592b3d-c373-48f3-a1b4-e75b8134afd6" volumeName="kubernetes.io/configmap/0c592b3d-c373-48f3-a1b4-e75b8134afd6-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782706 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067c1c4-cfc8-4358-9069-f24576389879" volumeName="kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 02 07:24:59.782708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782714 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782722 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-db" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782733 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e577f03-1ea2-4496-9adb-84a2dc4dc72f" volumeName="kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782741 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ebdbc81-5857-429e-9a08-2143afcdc097" volumeName="kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-tls" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782748 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067c1c4-cfc8-4358-9069-f24576389879" volumeName="kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-main-db" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782755 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92ebc3a2-3bc6-48d7-b78c-f2dd729c4759" volumeName="kubernetes.io/configmap/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782762 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1349fb3-19b0-4648-92e1-fcd2da2076bf" volumeName="kubernetes.io/projected/a1349fb3-19b0-4648-92e1-fcd2da2076bf-kube-api-access-5wkph" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782768 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e76535e7-a981-4b8a-9bcf-5e400d35c1d1" volumeName="kubernetes.io/secret/e76535e7-a981-4b8a-9bcf-5e400d35c1d1-cert" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782775 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-kube-api-access-n4hv6" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782782 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782789 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-web-config" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782796 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e577f03-1ea2-4496-9adb-84a2dc4dc72f" volumeName="kubernetes.io/empty-dir/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-audit-log" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782803 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0768e3b2-d790-4375-acf5-111424b1c5c3" volumeName="kubernetes.io/configmap/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782811 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b808d2a-56c8-477f-bca0-d9d69d711516" volumeName="kubernetes.io/projected/1b808d2a-56c8-477f-bca0-d9d69d711516-kube-api-access-d6mt7" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782818 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067c1c4-cfc8-4358-9069-f24576389879" volumeName="kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-cluster-tls-config" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782825 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782832 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782838 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77af1f03-6bc5-4dad-8646-eae4f5ae40f0" volumeName="kubernetes.io/configmap/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-iptables-alerter-script" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782845 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77af1f03-6bc5-4dad-8646-eae4f5ae40f0" volumeName="kubernetes.io/projected/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-kube-api-access-cg2fk" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782853 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b808d2a-56c8-477f-bca0-d9d69d711516" volumeName="kubernetes.io/configmap/1b808d2a-56c8-477f-bca0-d9d69d711516-config-volume" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782860 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b808d2a-56c8-477f-bca0-d9d69d711516" volumeName="kubernetes.io/empty-dir/1b808d2a-56c8-477f-bca0-d9d69d711516-tmp-dir" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782868 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eeba619-ec6d-4ec8-828b-2f409d53630d" volumeName="kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782875 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067c1c4-cfc8-4358-9069-f24576389879" volumeName="kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782882 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="973a9b8f-405e-4dc4-8322-4d12a152178d" volumeName="kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-ovnkube-config" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782889 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0" volumeName="kubernetes.io/projected/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0-kube-api-access-d6nkn" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782896 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85e9ffec-236a-4a8c-8dd1-7014e032e180" volumeName="kubernetes.io/configmap/85e9ffec-236a-4a8c-8dd1-7014e032e180-cni-binary-copy" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782903 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0768e3b2-d790-4375-acf5-111424b1c5c3" volumeName="kubernetes.io/configmap/0768e3b2-d790-4375-acf5-111424b1c5c3-metrics-client-ca" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782910 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eeba619-ec6d-4ec8-828b-2f409d53630d" volumeName="kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-cni-binary-copy" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782918 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067c1c4-cfc8-4358-9069-f24576389879" volumeName="kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-tls-assets" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782926 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c592b3d-c373-48f3-a1b4-e75b8134afd6" volumeName="kubernetes.io/empty-dir/0c592b3d-c373-48f3-a1b4-e75b8134afd6-data-volume" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782940 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c592b3d-c373-48f3-a1b4-e75b8134afd6" volumeName="kubernetes.io/secret/0c592b3d-c373-48f3-a1b4-e75b8134afd6-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782947 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e577f03-1ea2-4496-9adb-84a2dc4dc72f" volumeName="kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-secret-metrics-server-tls" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782954 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e577f03-1ea2-4496-9adb-84a2dc4dc72f" volumeName="kubernetes.io/configmap/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-metrics-server-audit-profiles" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782963 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="973a9b8f-405e-4dc4-8322-4d12a152178d" volumeName="kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-ovnkube-script-lib" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782976 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c592b3d-c373-48f3-a1b4-e75b8134afd6" volumeName="kubernetes.io/projected/0c592b3d-c373-48f3-a1b4-e75b8134afd6-kube-api-access-qjrb9" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782986 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92ebc3a2-3bc6-48d7-b78c-f2dd729c4759" volumeName="kubernetes.io/configmap/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-metrics-client-ca" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.782994 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ebdbc81-5857-429e-9a08-2143afcdc097" volumeName="kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783002 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783010 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783017 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71998a34-5575-4ae0-8892-2c9e88b1b396" volumeName="kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783024 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7" volumeName="kubernetes.io/secret/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783031 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" volumeName="kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-oauth-config" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783038 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config-out" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783045 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" volumeName="kubernetes.io/projected/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-kube-api-access-dbk9t" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783052 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ebdbc81-5857-429e-9a08-2143afcdc097" volumeName="kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783060 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067c1c4-cfc8-4358-9069-f24576389879" volumeName="kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-kube-api-access-h6mbb" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783067 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92ebc3a2-3bc6-48d7-b78c-f2dd729c4759" volumeName="kubernetes.io/secret/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783075 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="973a9b8f-405e-4dc4-8322-4d12a152178d" volumeName="kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-env-overrides" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783082 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e577f03-1ea2-4496-9adb-84a2dc4dc72f" volumeName="kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-client-ca-bundle" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783089 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ebdbc81-5857-429e-9a08-2143afcdc097" volumeName="kubernetes.io/configmap/2ebdbc81-5857-429e-9a08-2143afcdc097-metrics-client-ca" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783096 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92ebc3a2-3bc6-48d7-b78c-f2dd729c4759" volumeName="kubernetes.io/secret/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-tls" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783105 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0" volumeName="kubernetes.io/secret/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0-monitoring-plugin-cert" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783112 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0768e3b2-d790-4375-acf5-111424b1c5c3" volumeName="kubernetes.io/empty-dir/0768e3b2-d790-4375-acf5-111424b1c5c3-volume-directive-shadow" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783120 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3201efd6-0d33-4323-bab0-b76338d090ed" volumeName="kubernetes.io/projected/3201efd6-0d33-4323-bab0-b76338d090ed-kube-api-access-c7m65" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783127 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783135 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92ebc3a2-3bc6-48d7-b78c-f2dd729c4759" volumeName="kubernetes.io/empty-dir/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-textfile" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783142 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" volumeName="kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-serving-cert" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783150 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0768e3b2-d790-4375-acf5-111424b1c5c3" volumeName="kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783171 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eeba619-ec6d-4ec8-828b-2f409d53630d" volumeName="kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783183 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3201efd6-0d33-4323-bab0-b76338d090ed" volumeName="kubernetes.io/empty-dir/3201efd6-0d33-4323-bab0-b76338d090ed-tmp" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783195 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="973a9b8f-405e-4dc4-8322-4d12a152178d" volumeName="kubernetes.io/projected/973a9b8f-405e-4dc4-8322-4d12a152178d-kube-api-access-p6lln" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783208 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e229a07c-7415-4af9-a9eb-9e7984e0a510" volumeName="kubernetes.io/configmap/e229a07c-7415-4af9-a9eb-9e7984e0a510-serviceca" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783218 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3201efd6-0d33-4323-bab0-b76338d090ed" volumeName="kubernetes.io/empty-dir/3201efd6-0d33-4323-bab0-b76338d090ed-etc-tuned" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783225 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067c1c4-cfc8-4358-9069-f24576389879" volumeName="kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-web-config" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783233 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" volumeName="kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-service-ca" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783241 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6e945a4-9c6d-4310-902f-eff8fc826621" volumeName="kubernetes.io/configmap/d6e945a4-9c6d-4310-902f-eff8fc826621-konnectivity-ca" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783249 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0768e3b2-d790-4375-acf5-111424b1c5c3" volumeName="kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783256 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b808d2a-56c8-477f-bca0-d9d69d711516" volumeName="kubernetes.io/secret/1b808d2a-56c8-477f-bca0-d9d69d711516-metrics-tls" seLinuxMountContext="" Nov 02 07:24:59.783248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783280 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7" volumeName="kubernetes.io/configmap/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-metrics-client-ca" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783300 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" volumeName="kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-trusted-ca-bundle" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783312 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-tls-assets" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783333 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-metrics-client-certs" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783340 5899 factory.go:153] Registering CRI-O factory Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783351 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" volumeName="kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-config" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783356 5899 factory.go:223] Registration of the crio container factory successfully Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783362 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdb3f2d5-5e21-48c3-af74-eb8fd530046a" volumeName="kubernetes.io/empty-dir/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-tmp-dir" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783374 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067c1c4-cfc8-4358-9069-f24576389879" volumeName="kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783393 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" volumeName="kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-metrics-client-ca" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783413 5899 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783413 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7" volumeName="kubernetes.io/projected/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-kube-api-access-l7xw9" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783433 5899 factory.go:103] Registering Raw factory Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783456 5899 manager.go:1199] Started watching for new ooms in manager Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783782 5899 manager.go:319] Starting recovery of all containers Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.783432 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ebdbc81-5857-429e-9a08-2143afcdc097" volumeName="kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.784542 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067c1c4-cfc8-4358-9069-f24576389879" volumeName="kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.784569 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067c1c4-cfc8-4358-9069-f24576389879" volumeName="kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-config-volume" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.784584 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" volumeName="kubernetes.io/empty-dir/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-tmp" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.784597 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0768e3b2-d790-4375-acf5-111424b1c5c3" volumeName="kubernetes.io/projected/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-api-access-xrwkv" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.784609 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eeba619-ec6d-4ec8-828b-2f409d53630d" volumeName="kubernetes.io/projected/2eeba619-ec6d-4ec8-828b-2f409d53630d-kube-api-access-659fv" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.784621 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067c1c4-cfc8-4358-9069-f24576389879" volumeName="kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.784636 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92ebc3a2-3bc6-48d7-b78c-f2dd729c4759" volumeName="kubernetes.io/projected/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-kube-api-access-8fk87" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.784650 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef891d1c-0c5c-45b3-98eb-563324d8822f" volumeName="kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.784710 5899 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71998a34-5575-4ae0-8892-2c9e88b1b396" volumeName="kubernetes.io/projected/71998a34-5575-4ae0-8892-2c9e88b1b396-kube-api-access-pj2mr" seLinuxMountContext="" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.784726 5899 reconstruct.go:97] "Volume reconstruction finished" Nov 02 07:24:59.785857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.784736 5899 reconciler.go:29] "Reconciler: start to sync state" Nov 02 07:24:59.809551 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.809494 5899 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 07:24:59.811588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.811559 5899 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 07:24:59.811588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.811587 5899 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 07:24:59.811804 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.811610 5899 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 07:24:59.811804 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:24:59.811653 5899 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 07:24:59.828272 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.828235 5899 generic.go:356] "Generic (PLEG): container finished" podID="0c1a63f4a9c3f1c171863c543583e5bd" containerID="deda5e4a079b04917daf0b9178ea884cde825ec062a5af1f91efa2266ea9ba7e" exitCode=0 Nov 02 07:24:59.830130 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.829755 5899 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 07:24:59.830130 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.829943 5899 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 07:24:59.830130 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.830080 5899 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 07:24:59.833904 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.833874 5899 generic.go:356] "Generic (PLEG): container finished" podID="5067c1c4-cfc8-4358-9069-f24576389879" containerID="aa3e3d153f3a65cffc261804057dc9dbc532f39e6c0324199097adc4aea6bd2c" exitCode=0 Nov 02 07:24:59.840325 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.840301 5899 generic.go:356] "Generic (PLEG): container finished" podID="2eeba619-ec6d-4ec8-828b-2f409d53630d" containerID="e29557fff9cc0f97a93058bc808d2d41ba71765e5ead3562b13dac2ff6cb3204" exitCode=0 Nov 02 07:24:59.840325 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.840324 5899 generic.go:356] "Generic (PLEG): container finished" podID="2eeba619-ec6d-4ec8-828b-2f409d53630d" containerID="07d79a56b18f24da4ad9299ac8c27ba11f3e8a1345afc72e74c66de9d0e1c0da" exitCode=0 Nov 02 07:24:59.840471 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.840334 5899 generic.go:356] "Generic (PLEG): container finished" podID="2eeba619-ec6d-4ec8-828b-2f409d53630d" containerID="55f8cc03b956e487d1c3f597282c23f793a975f837ecf5fdb73fdfa4bff5e92d" exitCode=0 Nov 02 07:24:59.840471 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.840344 5899 generic.go:356] "Generic (PLEG): container finished" podID="2eeba619-ec6d-4ec8-828b-2f409d53630d" containerID="a834a5f9b4cc559ed1af1d146ee7e52d8421b295d02a8e656c500534f3222dbc" exitCode=0 Nov 02 07:24:59.840471 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.840352 5899 generic.go:356] "Generic (PLEG): container finished" podID="2eeba619-ec6d-4ec8-828b-2f409d53630d" containerID="bd0ea034e4ba55331c7fa4eee371b0ca47d7efe47f18a50cd0a3b0003c74fc61" exitCode=0 Nov 02 07:24:59.840471 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.840360 5899 generic.go:356] "Generic (PLEG): container finished" podID="2eeba619-ec6d-4ec8-828b-2f409d53630d" containerID="51fe96cb2c356fff6a19f73d3226016f9d910240087bcfd7bf776b33c2144b2a" exitCode=0 Nov 02 07:24:59.848136 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.848105 5899 generic.go:356] "Generic (PLEG): container finished" podID="92ebc3a2-3bc6-48d7-b78c-f2dd729c4759" containerID="16c3108888d8082597999b12a09c54255fbe301f956ad006067b5e8fa68f0093" exitCode=0 Nov 02 07:24:59.854846 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.854827 5899 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 07:24:59.855752 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.855731 5899 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 07:24:59.860282 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.860259 5899 generic.go:356] "Generic (PLEG): container finished" podID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerID="4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a" exitCode=0 Nov 02 07:24:59.910916 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.910852 5899 manager.go:324] Recovery completed Nov 02 07:24:59.911729 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:24:59.911710 5899 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 07:24:59.936857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.936827 5899 cpu_manager.go:226] "Starting" policy="none" Nov 02 07:24:59.936857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.936845 5899 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 07:24:59.936857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.936862 5899 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 07:24:59.937087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.936981 5899 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 07:24:59.937087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.936997 5899 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 07:24:59.937087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.937014 5899 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 07:24:59.937087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.937020 5899 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 07:24:59.937087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.937024 5899 policy_none.go:50] "Start" Nov 02 07:24:59.937087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.937033 5899 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 07:24:59.937087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.937042 5899 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 07:24:59.937325 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.937196 5899 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 07:24:59.937325 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.937206 5899 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 07:24:59.937325 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.937212 5899 policy_none.go:44] "Start" Nov 02 07:24:59.941941 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.941913 5899 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 07:24:59.944471 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.944459 5899 manager.go:341] "Starting Device Plugin manager" Nov 02 07:24:59.944976 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:24:59.944483 5899 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 07:24:59.945037 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.944979 5899 server.go:87] "Starting device plugin registration server" Nov 02 07:24:59.945235 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.945225 5899 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 07:24:59.945268 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.945236 5899 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 07:24:59.945355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.945332 5899 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 07:24:59.945510 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.945476 5899 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 07:24:59.945592 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.945518 5899 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 07:24:59.945592 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.945525 5899 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 07:24:59.945682 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.945575 5899 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T07:24:59.945507863Z","UUID":"6a827383-f2fb-404c-90f9-d28febe729f8","Handler":null,"Name":"","Endpoint":""} Nov 02 07:24:59.946982 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.946963 5899 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 07:24:59.947068 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:24:59.947000 5899 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 07:24:59.947263 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:24:59.947241 5899 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 07:25:00.045817 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.045779 5899 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 07:25:00.048518 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.048490 5899 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientMemory" Nov 02 07:25:00.048518 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.048522 5899 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 07:25:00.048649 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.048536 5899 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-6.ec2.internal" event="NodeHasSufficientPID" Nov 02 07:25:00.048649 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.048591 5899 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.084557 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.084535 5899 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.084629 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.084600 5899 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.112201 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.112143 5899 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal"] Nov 02 07:25:00.112813 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.112770 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" event={"ID":"0c1a63f4a9c3f1c171863c543583e5bd","Type":"ContainerStarted","Data":"5c77504387a2204768dbe9024e5a32f0eccba8fc3122097ec4612830c60aadc5"} Nov 02 07:25:00.112963 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.112822 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" event={"ID":"0c1a63f4a9c3f1c171863c543583e5bd","Type":"ContainerDied","Data":"deda5e4a079b04917daf0b9178ea884cde825ec062a5af1f91efa2266ea9ba7e"} Nov 02 07:25:00.112963 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.112846 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" event={"ID":"0c1a63f4a9c3f1c171863c543583e5bd","Type":"ContainerStarted","Data":"652ddf883f834360985f8af6947f12b2139cdbdff0c5cf725fcd714a869d2d2a"} Nov 02 07:25:00.112963 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.112860 5899 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.112963 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.112953 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" event={"ID":"cb0aa2bb77fd4bd0612d1dffa702ab46","Type":"ContainerStarted","Data":"a35e2331b24fadd9bdf4ffd0375d0689c18b7b57ce1f237c690c110bfb6eee39"} Nov 02 07:25:00.113215 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.112977 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" event={"ID":"cb0aa2bb77fd4bd0612d1dffa702ab46","Type":"ContainerStarted","Data":"82938c87d260f19af2189bdcbedcd4f36a06bb6d0d4719ea1975bc288ba4ae72"} Nov 02 07:25:00.113215 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.113074 5899 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.142203 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.142178 5899 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 07:25:00.142331 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:00.142218 5899 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-6.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.142781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.142766 5899 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 07:25:00.142817 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:00.142802 5899 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.231623 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.231597 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb0aa2bb77fd4bd0612d1dffa702ab46-config\") pod \"kube-apiserver-proxy-ip-10-0-135-6.ec2.internal\" (UID: \"cb0aa2bb77fd4bd0612d1dffa702ab46\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.231724 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.231630 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c1a63f4a9c3f1c171863c543583e5bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" (UID: \"0c1a63f4a9c3f1c171863c543583e5bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.231724 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.231649 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1a63f4a9c3f1c171863c543583e5bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" (UID: \"0c1a63f4a9c3f1c171863c543583e5bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.332207 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.332108 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c1a63f4a9c3f1c171863c543583e5bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" (UID: \"0c1a63f4a9c3f1c171863c543583e5bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.332324 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.332226 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1a63f4a9c3f1c171863c543583e5bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" (UID: \"0c1a63f4a9c3f1c171863c543583e5bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.332324 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.332243 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb0aa2bb77fd4bd0612d1dffa702ab46-config\") pod \"kube-apiserver-proxy-ip-10-0-135-6.ec2.internal\" (UID: \"cb0aa2bb77fd4bd0612d1dffa702ab46\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.332324 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.332151 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c1a63f4a9c3f1c171863c543583e5bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" (UID: \"0c1a63f4a9c3f1c171863c543583e5bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.332324 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.332311 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1a63f4a9c3f1c171863c543583e5bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" (UID: \"0c1a63f4a9c3f1c171863c543583e5bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.332457 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.332276 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb0aa2bb77fd4bd0612d1dffa702ab46-config\") pod \"kube-apiserver-proxy-ip-10-0-135-6.ec2.internal\" (UID: \"cb0aa2bb77fd4bd0612d1dffa702ab46\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.763841 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.763798 5899 apiserver.go:52] "Watching apiserver" Nov 02 07:25:00.820147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.820112 5899 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 07:25:00.820966 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.820941 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-v6hrz","openshift-cluster-node-tuning-operator/tuned-ts4xm","openshift-monitoring/alertmanager-main-0","kube-system/konnectivity-agent-5g4ck","openshift-multus/multus-additional-cni-plugins-qkcvh","openshift-network-operator/iptables-alerter-4hq85","openshift-ingress-canary/ingress-canary-9p24b","openshift-insights/insights-runtime-extractor-b486r","openshift-network-diagnostics/network-check-target-25f6d","openshift-dns/node-resolver-f6f9k","openshift-ovn-kubernetes/ovnkube-node-j948j","openshift-image-registry/node-ca-72ngx","openshift-monitoring/thanos-querier-6499784568-r4gwp","openshift-dns/dns-default-kp755","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal","openshift-monitoring/prometheus-k8s-0","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4","openshift-console/console-7cc9dd686b-z95lj","openshift-monitoring/node-exporter-kk7sz","openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l","openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh","openshift-multus/multus-pm58s","openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh","kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal","openshift-monitoring/metrics-server-895bcdc89-gxp7k","kube-system/global-pull-secret-syncer-9ztwm"] Nov 02 07:25:00.837719 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.837682 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 07:25:00.837719 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.837710 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hdb4w\"" Nov 02 07:25:00.837719 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.837718 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 07:25:00.837947 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.837710 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 07:25:00.837947 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.837758 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfpr4\"" Nov 02 07:25:00.837947 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.837762 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 07:25:00.843356 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843327 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 07:25:00.843356 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843348 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 07:25:00.843615 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843346 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zwm44\"" Nov 02 07:25:00.843615 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843352 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 07:25:00.843615 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843428 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lptml\"" Nov 02 07:25:00.843615 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843439 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 07:25:00.843615 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843446 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c9gk6\"" Nov 02 07:25:00.843615 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843462 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 07:25:00.843615 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843591 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 07:25:00.843862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843621 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 07:25:00.843862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843629 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 07:25:00.843862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843651 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 07:25:00.843862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843656 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 07:25:00.843862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843667 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 07:25:00.843862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843686 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 07:25:00.843862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843689 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 07:25:00.843862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.843679 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7nffm\"" Nov 02 07:25:00.850736 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.850717 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 07:25:00.888194 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.885820 5899 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 07:25:00.889490 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.889470 5899 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.889673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.889658 5899 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.913872 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.913848 5899 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 07:25:00.914003 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:00.913901 5899 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-6.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.914516 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.914500 5899 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 07:25:00.914617 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:00.914537 5899 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-6.ec2.internal" Nov 02 07:25:00.934686 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.934653 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.934686 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.934694 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-registration-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:00.934857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.934711 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-cni-multus\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.934857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.934729 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-host\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.934857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.934746 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-systemd-units\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.934857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.934793 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:00.935051 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.934908 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-client-ca-bundle\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.935051 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.934976 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:00.935051 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.934973 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-web-config\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.935051 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935009 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.935051 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935027 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysconfig\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.935051 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935043 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-sys\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.935355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935060 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qfz\" (UniqueName: \"kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz\") pod \"network-check-target-25f6d\" (UID: \"ef891d1c-0c5c-45b3-98eb-563324d8822f\") " pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:25:00.935355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935111 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mxbc\" (UniqueName: \"kubernetes.io/projected/2ebdbc81-5857-429e-9a08-2143afcdc097-kube-api-access-2mxbc\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.935355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935145 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-oauth-config\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.935355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935195 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-service-ca\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.935355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935225 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6mbb\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-kube-api-access-h6mbb\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.935355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935253 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b808d2a-56c8-477f-bca0-d9d69d711516-tmp-dir\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:25:00.935355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935276 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.935355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935302 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-config-out\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.935355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935327 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.935355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935355 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935358 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b808d2a-56c8-477f-bca0-d9d69d711516-tmp-dir\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935383 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-cnibin\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935404 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-config-out\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935445 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-metrics-server-audit-profiles\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935472 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935499 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935527 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935571 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-audit-log\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935598 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnzm\" (UniqueName: \"kubernetes.io/projected/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-kube-api-access-djnzm\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935617 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-slash\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935640 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-metrics-client-ca\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935664 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-hostroot\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935685 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-tmp-dir\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935685 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-audit-log\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935717 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6lln\" (UniqueName: \"kubernetes.io/projected/973a9b8f-405e-4dc4-8322-4d12a152178d-kube-api-access-p6lln\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935748 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0768e3b2-d790-4375-acf5-111424b1c5c3-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:25:00.935758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935766 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-tmp-dir\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935776 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-system-cni-dir\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935803 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935812 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0768e3b2-d790-4375-acf5-111424b1c5c3-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935834 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hv6\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-kube-api-access-n4hv6\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935926 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-sys-fs\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935943 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-kubernetes\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.935986 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6nkn\" (UniqueName: \"kubernetes.io/projected/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0-kube-api-access-d6nkn\") pod \"monitoring-plugin-77f4f76594-xfn8l\" (UID: \"b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936037 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-host-slash\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936037 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936119 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c592b3d-c373-48f3-a1b4-e75b8134afd6-data-volume\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936180 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936214 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936221 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c592b3d-c373-48f3-a1b4-e75b8134afd6-data-volume\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936237 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-cnibin\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936262 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936287 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-etc-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936393 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c592b3d-c373-48f3-a1b4-e75b8134afd6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936463 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-sys\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936515 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-root\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.936546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936551 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936599 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-kubelet\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936627 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-lib-modules\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936657 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936706 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6mt7\" (UniqueName: \"kubernetes.io/projected/1b808d2a-56c8-477f-bca0-d9d69d711516-kube-api-access-d6mt7\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936734 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-tmp\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936765 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936792 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvtjs\" (UniqueName: \"kubernetes.io/projected/85e9ffec-236a-4a8c-8dd1-7014e032e180-kube-api-access-lvtjs\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936818 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7m65\" (UniqueName: \"kubernetes.io/projected/3201efd6-0d33-4323-bab0-b76338d090ed-kube-api-access-c7m65\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936844 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936866 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-tmp\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936871 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-kubelet\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936922 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-systemd\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936961 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-ovn\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.936994 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-secret-metrics-server-tls\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937047 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937067 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wkph\" (UniqueName: \"kubernetes.io/projected/a1349fb3-19b0-4648-92e1-fcd2da2076bf-kube-api-access-5wkph\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937087 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-system-cni-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937101 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysctl-d\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937119 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937187 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937221 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-env-overrides\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937245 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6e945a4-9c6d-4310-902f-eff8fc826621-agent-certs\") pod \"konnectivity-agent-5g4ck\" (UID: \"d6e945a4-9c6d-4310-902f-eff8fc826621\") " pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:25:00.937279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937271 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937296 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-web-config\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937322 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-cni-netd\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937349 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c592b3d-c373-48f3-a1b4-e75b8134afd6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937378 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e76535e7-a981-4b8a-9bcf-5e400d35c1d1-cert\") pod \"ingress-canary-9p24b\" (UID: \"e76535e7-a981-4b8a-9bcf-5e400d35c1d1\") " pod="openshift-ingress-canary/ingress-canary-9p24b" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937403 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zx4h\" (UniqueName: \"kubernetes.io/projected/e76535e7-a981-4b8a-9bcf-5e400d35c1d1-kube-api-access-4zx4h\") pod \"ingress-canary-9p24b\" (UID: \"e76535e7-a981-4b8a-9bcf-5e400d35c1d1\") " pod="openshift-ingress-canary/ingress-canary-9p24b" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937430 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937439 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6e945a4-9c6d-4310-902f-eff8fc826621-agent-certs\") pod \"konnectivity-agent-5g4ck\" (UID: \"d6e945a4-9c6d-4310-902f-eff8fc826621\") " pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937538 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fk87\" (UniqueName: \"kubernetes.io/projected/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-kube-api-access-8fk87\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937563 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937582 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-netns\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937610 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0768e3b2-d790-4375-acf5-111424b1c5c3-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937714 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-tls\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937747 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937775 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ebdbc81-5857-429e-9a08-2143afcdc097-tmp\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937801 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937826 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937852 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937867 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ebdbc81-5857-429e-9a08-2143afcdc097-tmp\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937879 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-accelerators-collector-config\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937889 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937914 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hrz7\" (UniqueName: \"kubernetes.io/projected/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-kube-api-access-5hrz7\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937919 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.937958 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/253a4509-48e5-48c2-8c13-2dc361e5763f-dbus\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938047 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-secret-metrics-server-client-certs\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938070 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-config\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938100 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-oauth-serving-cert\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938119 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2eeba619-ec6d-4ec8-828b-2f409d53630d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:00.938141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938124 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbk9t\" (UniqueName: \"kubernetes.io/projected/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-kube-api-access-dbk9t\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938183 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938194 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85e9ffec-236a-4a8c-8dd1-7014e032e180-cni-binary-copy\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938216 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938232 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938235 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938295 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938332 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938372 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938404 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-modprobe-d\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938431 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-node-log\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938421 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85e9ffec-236a-4a8c-8dd1-7014e032e180-cni-binary-copy\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938458 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-device-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938478 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-socket-dir-parent\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938536 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/253a4509-48e5-48c2-8c13-2dc361e5763f-kubelet-config\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938577 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-ovnkube-script-lib\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938638 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938647 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-wtmp\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938716 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-iptables-alerter-script\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938731 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938767 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj2mr\" (UniqueName: \"kubernetes.io/projected/71998a34-5575-4ae0-8892-2c9e88b1b396-kube-api-access-pj2mr\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938793 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938802 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938834 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-multus-certs\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938849 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938862 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-etc-kubernetes\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938882 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938891 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysctl-conf\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938932 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xftv\" (UniqueName: \"kubernetes.io/projected/e229a07c-7415-4af9-a9eb-9e7984e0a510-kube-api-access-2xftv\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.938963 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-cni-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939021 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939022 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-iptables-alerter-script\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939055 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939083 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-659fv\" (UniqueName: \"kubernetes.io/projected/2eeba619-ec6d-4ec8-828b-2f409d53630d-kube-api-access-659fv\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939118 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e229a07c-7415-4af9-a9eb-9e7984e0a510-host\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939143 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-tls\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939196 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3201efd6-0d33-4323-bab0-b76338d090ed-tmp\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939238 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939272 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939292 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939303 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939324 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-run-ovn-kubernetes\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939358 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3201efd6-0d33-4323-bab0-b76338d090ed-tmp\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939363 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrwkv\" (UniqueName: \"kubernetes.io/projected/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-api-access-xrwkv\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:25:00.939404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939426 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b808d2a-56c8-477f-bca0-d9d69d711516-config-volume\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939479 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939489 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config-out\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939520 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939554 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-socket-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939584 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-k8s-cni-cncf-io\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939592 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939608 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939638 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939609 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-run\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939681 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3201efd6-0d33-4323-bab0-b76338d090ed-etc-tuned\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939694 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config-out\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939709 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939743 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939753 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-67pc5\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939774 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-log-socket\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939788 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3201efd6-0d33-4323-bab0-b76338d090ed-etc-tuned\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939811 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939841 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-cni-bin\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939864 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939815 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-aovfv7t8ki78k\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939875 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-hosts-file\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939918 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-var-lib-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939940 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tn89g\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939974 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-serving-cert\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.939997 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940021 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940039 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940077 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940132 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-cni-bin\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940146 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940198 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-ovnkube-config\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940231 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xfn8l\" (UID: \"b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940236 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940243 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940259 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940268 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940286 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940301 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940360 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940421 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-grpc-tls\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940440 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7x2d2\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940429 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940458 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940489 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6e945a4-9c6d-4310-902f-eff8fc826621-konnectivity-ca\") pod \"konnectivity-agent-5g4ck\" (UID: \"d6e945a4-9c6d-4310-902f-eff8fc826621\") " pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940495 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940461 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940513 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940517 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-var-lib-kubelet\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940545 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-trusted-ca-bundle\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940573 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940621 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-config-volume\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940644 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2fk\" (UniqueName: \"kubernetes.io/projected/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-kube-api-access-cg2fk\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940661 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-daemon-config\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940693 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-grpc-tls\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940712 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6e945a4-9c6d-4310-902f-eff8fc826621-konnectivity-ca\") pod \"konnectivity-agent-5g4ck\" (UID: \"d6e945a4-9c6d-4310-902f-eff8fc826621\") " pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940772 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940805 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7xw9\" (UniqueName: \"kubernetes.io/projected/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-kube-api-access-l7xw9\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940808 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940827 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-config-volume\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940836 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-textfile\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940866 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940879 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940895 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940917 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940926 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940967 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rhkvn\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940924 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b808d2a-56c8-477f-bca0-d9d69d711516-metrics-tls\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940987 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-htl44\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940999 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941010 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941014 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941019 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940971 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940974 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941049 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-textfile\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941055 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941076 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941089 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-32bjlgfpvos9j\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941095 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjrb9\" (UniqueName: \"kubernetes.io/projected/0c592b3d-c373-48f3-a1b4-e75b8134afd6-kube-api-access-qjrb9\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.940897 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941124 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941132 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941133 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jcgqg\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941179 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-szw6v\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941202 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941208 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941215 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7q84k\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941225 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941241 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941257 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-45mkt\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941258 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941266 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941126 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ebdbc81-5857-429e-9a08-2143afcdc097-metrics-client-ca\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941264 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941305 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941311 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tc9dv\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941304 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941310 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e229a07c-7415-4af9-a9eb-9e7984e0a510-serviceca\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941330 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941332 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941275 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941339 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w2bbn\"" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941358 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941387 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:25:00.941355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941415 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941421 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941478 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941498 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941511 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-conf-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941551 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-systemd\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941578 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-run-netns\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941623 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/973a9b8f-405e-4dc4-8322-4d12a152178d-ovn-node-metrics-cert\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941659 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c592b3d-c373-48f3-a1b4-e75b8134afd6-crio-socket\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941685 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941689 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/253a4509-48e5-48c2-8c13-2dc361e5763f-original-pull-secret\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941772 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e229a07c-7415-4af9-a9eb-9e7984e0a510-serviceca\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.941839 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/973a9b8f-405e-4dc4-8322-4d12a152178d-ovn-node-metrics-cert\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.943296 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhg4q\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.943587 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.943620 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6xxpk\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.943631 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.943643 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.943657 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.943668 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jc9nv\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.943698 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.943698 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.943724 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.943730 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zwd5q\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.943754 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-scd8z\"" Nov 02 07:25:00.944287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.943791 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lsjmf\"" Nov 02 07:25:00.946280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.946131 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.946280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.946180 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-web-config\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.946280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.946195 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.946280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.946206 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-service-ca\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.946280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.946238 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.946280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.946249 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:25:00.946280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.946239 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.946530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.946295 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-oauth-config\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.946530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.946299 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-metrics-server-audit-profiles\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.946530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.946336 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.946530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.946399 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.946530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.946513 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.946674 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.946600 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c592b3d-c373-48f3-a1b4-e75b8134afd6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:25:00.947650 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947482 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.947650 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947572 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-web-config\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.947650 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947592 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.947650 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947599 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.947650 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947572 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.947650 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947611 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-env-overrides\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.947650 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947626 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.947650 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947595 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.948043 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947671 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c592b3d-c373-48f3-a1b4-e75b8134afd6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:25:00.948043 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947678 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:25:00.948043 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947735 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e76535e7-a981-4b8a-9bcf-5e400d35c1d1-cert\") pod \"ingress-canary-9p24b\" (UID: \"e76535e7-a981-4b8a-9bcf-5e400d35c1d1\") " pod="openshift-ingress-canary/ingress-canary-9p24b" Nov 02 07:25:00.948043 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947807 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.948043 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947934 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-tls\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.948043 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.947959 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.948231 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.948071 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-accelerators-collector-config\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.948231 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.948085 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.948828 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.948809 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2ebdbc81-5857-429e-9a08-2143afcdc097-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.948906 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.948884 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-oauth-serving-cert\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.948906 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.948891 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.949006 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.948907 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-secret-metrics-server-client-certs\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.949006 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.948924 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:25:00.949006 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.948909 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-config\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.949133 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.949043 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.950060 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.950043 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b808d2a-56c8-477f-bca0-d9d69d711516-config-volume\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:25:00.950117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.950078 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-tls\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.950117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.950097 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.950204 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.950147 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:25:00.951120 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.951070 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.951120 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.951077 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 07:25:00.951120 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.951096 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-daemon-config\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.951282 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.951127 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-ovnkube-config\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.951282 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.951129 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xfn8l\" (UID: \"b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:25:00.951282 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.951129 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ln5vb\"" Nov 02 07:25:00.951282 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.951181 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71998a34-5575-4ae0-8892-2c9e88b1b396-metrics-certs\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:25:00.951282 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.951196 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b808d2a-56c8-477f-bca0-d9d69d711516-metrics-tls\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:25:00.951567 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.951551 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 07:25:00.952644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.952627 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 07:25:00.952695 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.952661 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 07:25:00.952695 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.952682 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7n5qmumkac75f\"" Nov 02 07:25:00.953435 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.953420 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 07:25:00.955292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.955275 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-client-ca-bundle\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.955981 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.955966 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-metrics-client-ca\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.956635 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.956616 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:25:00.957276 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.957253 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-secret-metrics-server-tls\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.957551 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.957535 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 07:25:00.957700 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.957682 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 07:25:00.957860 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.957845 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0768e3b2-d790-4375-acf5-111424b1c5c3-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:25:00.957909 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.957869 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 07:25:00.957909 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.957869 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 07:25:00.962834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.959210 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/973a9b8f-405e-4dc4-8322-4d12a152178d-ovnkube-script-lib\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.962834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.959652 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.962834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.959698 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6nkn\" (UniqueName: \"kubernetes.io/projected/b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0-kube-api-access-d6nkn\") pod \"monitoring-plugin-77f4f76594-xfn8l\" (UID: \"b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:25:00.962834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.959915 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.962834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.960501 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-serving-cert\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.962834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.961145 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7m65\" (UniqueName: \"kubernetes.io/projected/3201efd6-0d33-4323-bab0-b76338d090ed-kube-api-access-c7m65\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:00.962834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.961741 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6mt7\" (UniqueName: \"kubernetes.io/projected/1b808d2a-56c8-477f-bca0-d9d69d711516-kube-api-access-d6mt7\") pod \"dns-default-kp755\" (UID: \"1b808d2a-56c8-477f-bca0-d9d69d711516\") " pod="openshift-dns/dns-default-kp755" Nov 02 07:25:00.962834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.962516 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8qfz\" (UniqueName: \"kubernetes.io/projected/ef891d1c-0c5c-45b3-98eb-563324d8822f-kube-api-access-w8qfz\") pod \"network-check-target-25f6d\" (UID: \"ef891d1c-0c5c-45b3-98eb-563324d8822f\") " pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:25:00.963280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.963037 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:25:00.963280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.963042 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.963280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.963048 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-trusted-ca-bundle\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.963280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.963054 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ebdbc81-5857-429e-9a08-2143afcdc097-metrics-client-ca\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.963803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.963781 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6lln\" (UniqueName: \"kubernetes.io/projected/973a9b8f-405e-4dc4-8322-4d12a152178d-kube-api-access-p6lln\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:00.963898 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.963876 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mxbc\" (UniqueName: \"kubernetes.io/projected/2ebdbc81-5857-429e-9a08-2143afcdc097-kube-api-access-2mxbc\") pod \"thanos-querier-6499784568-r4gwp\" (UID: \"2ebdbc81-5857-429e-9a08-2143afcdc097\") " pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:00.964076 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.964056 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hv6\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-kube-api-access-n4hv6\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.964124 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.964103 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hrz7\" (UniqueName: \"kubernetes.io/projected/0e577f03-1ea2-4496-9adb-84a2dc4dc72f-kube-api-access-5hrz7\") pod \"metrics-server-895bcdc89-gxp7k\" (UID: \"0e577f03-1ea2-4496-9adb-84a2dc4dc72f\") " pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:00.964882 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.964858 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj2mr\" (UniqueName: \"kubernetes.io/projected/71998a34-5575-4ae0-8892-2c9e88b1b396-kube-api-access-pj2mr\") pod \"network-metrics-daemon-v6hrz\" (UID: \"71998a34-5575-4ae0-8892-2c9e88b1b396\") " pod="openshift-multus/network-metrics-daemon-v6hrz" Nov 02 07:25:00.965007 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.964986 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnzm\" (UniqueName: \"kubernetes.io/projected/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-kube-api-access-djnzm\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:25:00.965074 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.965000 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvtjs\" (UniqueName: \"kubernetes.io/projected/85e9ffec-236a-4a8c-8dd1-7014e032e180-kube-api-access-lvtjs\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:00.965074 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.965016 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wkph\" (UniqueName: \"kubernetes.io/projected/a1349fb3-19b0-4648-92e1-fcd2da2076bf-kube-api-access-5wkph\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:00.965074 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.965029 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fk87\" (UniqueName: \"kubernetes.io/projected/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-kube-api-access-8fk87\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:00.965256 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.965194 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.965332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.965315 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6mbb\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-kube-api-access-h6mbb\") pod \"alertmanager-main-0\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:00.965755 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.965736 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zx4h\" (UniqueName: \"kubernetes.io/projected/e76535e7-a981-4b8a-9bcf-5e400d35c1d1-kube-api-access-4zx4h\") pod \"ingress-canary-9p24b\" (UID: \"e76535e7-a981-4b8a-9bcf-5e400d35c1d1\") " pod="openshift-ingress-canary/ingress-canary-9p24b" Nov 02 07:25:00.967049 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.967028 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:00.967106 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.967095 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xftv\" (UniqueName: \"kubernetes.io/projected/e229a07c-7415-4af9-a9eb-9e7984e0a510-kube-api-access-2xftv\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:25:00.969310 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.969292 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbk9t\" (UniqueName: \"kubernetes.io/projected/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-kube-api-access-dbk9t\") pod \"console-7cc9dd686b-z95lj\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:00.969383 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.969330 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7xw9\" (UniqueName: \"kubernetes.io/projected/8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7-kube-api-access-l7xw9\") pod \"openshift-state-metrics-7bd7489957-5zkrh\" (UID: \"8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5zkrh" Nov 02 07:25:00.969418 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.969388 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-659fv\" (UniqueName: \"kubernetes.io/projected/2eeba619-ec6d-4ec8-828b-2f409d53630d-kube-api-access-659fv\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:00.969658 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.969642 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjrb9\" (UniqueName: \"kubernetes.io/projected/0c592b3d-c373-48f3-a1b4-e75b8134afd6-kube-api-access-qjrb9\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:25:00.970566 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.970546 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2fk\" (UniqueName: \"kubernetes.io/projected/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-kube-api-access-cg2fk\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:25:00.971417 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:00.971401 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrwkv\" (UniqueName: \"kubernetes.io/projected/0768e3b2-d790-4375-acf5-111424b1c5c3-kube-api-access-xrwkv\") pod \"kube-state-metrics-5dc8d7d89b-sbrnh\" (UID: \"0768e3b2-d790-4375-acf5-111424b1c5c3\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sbrnh" Nov 02 07:25:01.042336 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042267 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-var-lib-kubelet\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.042336 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042299 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:01.042336 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042317 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-conf-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.042530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042333 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-systemd\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.042530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042389 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-run-netns\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.042530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042408 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:01.042530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042406 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-var-lib-kubelet\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.042530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042423 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c592b3d-c373-48f3-a1b4-e75b8134afd6-crio-socket\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:25:01.042530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042406 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-conf-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.042530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042453 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-systemd\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.042530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042461 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-registration-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:01.042530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042467 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c592b3d-c373-48f3-a1b4-e75b8134afd6-crio-socket\") pod \"insights-runtime-extractor-b486r\" (UID: \"0c592b3d-c373-48f3-a1b4-e75b8134afd6\") " pod="openshift-insights/insights-runtime-extractor-b486r" Nov 02 07:25:01.042530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042443 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-run-netns\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.042530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042503 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-cni-multus\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.042530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042505 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-registration-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042561 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-cni-multus\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042585 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-host\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042604 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-systemd-units\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042634 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysconfig\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042642 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-systemd-units\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042657 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-sys\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042653 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-host\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042695 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysconfig\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042697 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-cnibin\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042728 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-cnibin\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042736 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042766 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-slash\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042780 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-hostroot\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042795 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-system-cni-dir\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042794 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042819 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-sys-fs\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042821 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-slash\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042742 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-sys\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042844 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-system-cni-dir\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042842 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-kubernetes\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042857 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-hostroot\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042872 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-kubernetes\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042875 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-host-slash\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042902 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-cnibin\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042870 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-sys-fs\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042917 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-etc-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042932 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77af1f03-6bc5-4dad-8646-eae4f5ae40f0-host-slash\") pod \"iptables-alerter-4hq85\" (UID: \"77af1f03-6bc5-4dad-8646-eae4f5ae40f0\") " pod="openshift-network-operator/iptables-alerter-4hq85" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042934 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-etc-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042954 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-sys\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042970 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-cnibin\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042971 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-root\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042991 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-root\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.042992 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-kubelet\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043009 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-kubelet\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.042990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043011 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-lib-modules\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043034 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-sys\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043042 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-kubelet\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043080 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-kubelet\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043086 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-lib-modules\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043105 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-systemd\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043123 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-ovn\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043142 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-system-cni-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043182 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysctl-d\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043189 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-ovn\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043205 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043244 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043226 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-systemd\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043250 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-system-cni-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043262 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-cni-netd\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043247 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-run-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043282 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-netns\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043287 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysctl-d\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043304 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-netns\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043305 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-cni-netd\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043282 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043305 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043346 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eeba619-ec6d-4ec8-828b-2f409d53630d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkcvh\" (UID: \"2eeba619-ec6d-4ec8-828b-2f409d53630d\") " pod="openshift-multus/multus-additional-cni-plugins-qkcvh" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043366 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/253a4509-48e5-48c2-8c13-2dc361e5763f-dbus\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043389 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-modprobe-d\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043403 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-node-log\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043419 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-device-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043444 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-socket-dir-parent\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043457 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-node-log\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043478 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/253a4509-48e5-48c2-8c13-2dc361e5763f-kubelet-config\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043483 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-device-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043476 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-modprobe-d\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043499 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-socket-dir-parent\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043511 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/253a4509-48e5-48c2-8c13-2dc361e5763f-dbus\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043512 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-wtmp\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043534 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/253a4509-48e5-48c2-8c13-2dc361e5763f-kubelet-config\") pod \"global-pull-secret-syncer-9ztwm\" (UID: \"253a4509-48e5-48c2-8c13-2dc361e5763f\") " pod="kube-system/global-pull-secret-syncer-9ztwm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043542 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-multus-certs\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043554 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92ebc3a2-3bc6-48d7-b78c-f2dd729c4759-node-exporter-wtmp\") pod \"node-exporter-kk7sz\" (UID: \"92ebc3a2-3bc6-48d7-b78c-f2dd729c4759\") " pod="openshift-monitoring/node-exporter-kk7sz" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043558 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-etc-kubernetes\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043579 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-etc-kubernetes\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043591 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-multus-certs\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043591 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysctl-conf\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043619 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-cni-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043643 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e229a07c-7415-4af9-a9eb-9e7984e0a510-host\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043671 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-run-ovn-kubernetes\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043690 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-multus-cni-dir\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043694 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-etc-sysctl-conf\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043695 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e229a07c-7415-4af9-a9eb-9e7984e0a510-host\") pod \"node-ca-72ngx\" (UID: \"e229a07c-7415-4af9-a9eb-9e7984e0a510\") " pod="openshift-image-registry/node-ca-72ngx" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043698 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-socket-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043726 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-run-ovn-kubernetes\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043742 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-k8s-cni-cncf-io\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043762 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-run\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043777 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-log-socket\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043781 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-run-k8s-cni-cncf-io\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043790 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-cni-bin\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043813 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-hosts-file\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043743 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1349fb3-19b0-4648-92e1-fcd2da2076bf-socket-dir\") pod \"aws-ebs-csi-driver-node-twnp4\" (UID: \"a1349fb3-19b0-4648-92e1-fcd2da2076bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twnp4" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043838 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3201efd6-0d33-4323-bab0-b76338d090ed-run\") pod \"tuned-ts4xm\" (UID: \"3201efd6-0d33-4323-bab0-b76338d090ed\") " pod="openshift-cluster-node-tuning-operator/tuned-ts4xm" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043836 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-log-socket\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043838 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-var-lib-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043862 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fdb3f2d5-5e21-48c3-af74-eb8fd530046a-hosts-file\") pod \"node-resolver-f6f9k\" (UID: \"fdb3f2d5-5e21-48c3-af74-eb8fd530046a\") " pod="openshift-dns/node-resolver-f6f9k" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043861 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85e9ffec-236a-4a8c-8dd1-7014e032e180-host-var-lib-cni-bin\") pod \"multus-pm58s\" (UID: \"85e9ffec-236a-4a8c-8dd1-7014e032e180\") " pod="openshift-multus/multus-pm58s" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043868 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-var-lib-openvswitch\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043884 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-cni-bin\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.044269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.043931 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/973a9b8f-405e-4dc4-8322-4d12a152178d-host-cni-bin\") pod \"ovnkube-node-j948j\" (UID: \"973a9b8f-405e-4dc4-8322-4d12a152178d\") " pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:01.575467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:01.575436 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kp755" Nov 02 07:25:02.089711 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:02.089681 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:03.265079 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:03.265046 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:04.157732 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:04.157700 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:04.157927 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:04.157908 5899 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 07:25:05.457209 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:05.457177 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6499784568-r4gwp" Nov 02 07:25:05.479387 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:05.479353 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:25:09.749190 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:09.749146 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j948j" Nov 02 07:25:10.075200 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:10.075115 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:25:10.626253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:10.626215 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xfn8l" Nov 02 07:25:13.139248 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:13.139213 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:25:13.933416 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:13.933388 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-895bcdc89-gxp7k" Nov 02 07:25:14.083546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:14.083507 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:25:14.217129 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:14.217047 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:25:18.758088 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:18.758049 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:25:18.891893 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:18.891860 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:25:20.067033 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.066999 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 07:25:20.067464 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.067434 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="kube-rbac-proxy-metric" containerID="cri-o://d108b1ba1274ac67203c71a5c6b5133fdf8402b0707d3229b48bba80d503ddcd" gracePeriod=120 Nov 02 07:25:20.067538 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.067436 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="alertmanager" containerID="cri-o://f63e67969306d93adfebea905e48f96b10b8be605fa957d8bebded6f85c556d0" gracePeriod=120 Nov 02 07:25:20.067538 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.067436 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="kube-rbac-proxy-web" containerID="cri-o://27128c6aee92197baab9a76c99d441338d6bbdde3c3f2cc99f4fc5493d6f8462" gracePeriod=120 Nov 02 07:25:20.067646 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.067500 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="prom-label-proxy" containerID="cri-o://e7d6f87053a8f2d472dd83cc9d778f0bd1bfb30652c9452f693cbbcfade68c5d" gracePeriod=120 Nov 02 07:25:20.067646 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.067518 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="config-reloader" containerID="cri-o://7dc06196250deaccc91c194e66dc0d821d6d9ec5977a08579c9fae4e5afc2d5a" gracePeriod=120 Nov 02 07:25:20.067646 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.067462 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="kube-rbac-proxy" containerID="cri-o://0f7556c644d1625c07cfb8660f8d413bb604057329cb98f66b9c3f01007c1efc" gracePeriod=120 Nov 02 07:25:20.965464 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.965429 5899 generic.go:356] "Generic (PLEG): container finished" podID="5067c1c4-cfc8-4358-9069-f24576389879" containerID="e7d6f87053a8f2d472dd83cc9d778f0bd1bfb30652c9452f693cbbcfade68c5d" exitCode=0 Nov 02 07:25:20.965464 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.965454 5899 generic.go:356] "Generic (PLEG): container finished" podID="5067c1c4-cfc8-4358-9069-f24576389879" containerID="d108b1ba1274ac67203c71a5c6b5133fdf8402b0707d3229b48bba80d503ddcd" exitCode=0 Nov 02 07:25:20.965464 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.965460 5899 generic.go:356] "Generic (PLEG): container finished" podID="5067c1c4-cfc8-4358-9069-f24576389879" containerID="0f7556c644d1625c07cfb8660f8d413bb604057329cb98f66b9c3f01007c1efc" exitCode=0 Nov 02 07:25:20.965464 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.965465 5899 generic.go:356] "Generic (PLEG): container finished" podID="5067c1c4-cfc8-4358-9069-f24576389879" containerID="7dc06196250deaccc91c194e66dc0d821d6d9ec5977a08579c9fae4e5afc2d5a" exitCode=0 Nov 02 07:25:20.965464 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.965470 5899 generic.go:356] "Generic (PLEG): container finished" podID="5067c1c4-cfc8-4358-9069-f24576389879" containerID="f63e67969306d93adfebea905e48f96b10b8be605fa957d8bebded6f85c556d0" exitCode=0 Nov 02 07:25:20.965742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.965502 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerDied","Data":"e7d6f87053a8f2d472dd83cc9d778f0bd1bfb30652c9452f693cbbcfade68c5d"} Nov 02 07:25:20.965742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.965589 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerDied","Data":"d108b1ba1274ac67203c71a5c6b5133fdf8402b0707d3229b48bba80d503ddcd"} Nov 02 07:25:20.965742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.965615 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerDied","Data":"0f7556c644d1625c07cfb8660f8d413bb604057329cb98f66b9c3f01007c1efc"} Nov 02 07:25:20.965742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.965639 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerDied","Data":"7dc06196250deaccc91c194e66dc0d821d6d9ec5977a08579c9fae4e5afc2d5a"} Nov 02 07:25:20.965742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:20.965660 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerDied","Data":"f63e67969306d93adfebea905e48f96b10b8be605fa957d8bebded6f85c556d0"} Nov 02 07:25:21.311978 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.311954 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:21.323338 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323313 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-main-db\") pod \"5067c1c4-cfc8-4358-9069-f24576389879\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " Nov 02 07:25:21.323459 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323347 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-config-volume\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-config-volume\") pod \"5067c1c4-cfc8-4358-9069-f24576389879\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " Nov 02 07:25:21.323459 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323372 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-config-out\") pod \"5067c1c4-cfc8-4358-9069-f24576389879\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " Nov 02 07:25:21.323459 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323390 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-web\") pod \"5067c1c4-cfc8-4358-9069-f24576389879\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " Nov 02 07:25:21.323459 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323415 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-tls-assets\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-tls-assets\") pod \"5067c1c4-cfc8-4358-9069-f24576389879\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " Nov 02 07:25:21.323673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323569 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-web-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-web-config\") pod \"5067c1c4-cfc8-4358-9069-f24576389879\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " Nov 02 07:25:21.323673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323633 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-metrics-client-ca\") pod \"5067c1c4-cfc8-4358-9069-f24576389879\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " Nov 02 07:25:21.323772 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323690 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-main-db" pod "5067c1c4-cfc8-4358-9069-f24576389879" (UID: "5067c1c4-cfc8-4358-9069-f24576389879"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:25:21.323772 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323712 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-trusted-ca-bundle\") pod \"5067c1c4-cfc8-4358-9069-f24576389879\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " Nov 02 07:25:21.323885 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323774 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-cluster-tls-config\") pod \"5067c1c4-cfc8-4358-9069-f24576389879\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " Nov 02 07:25:21.323885 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323809 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-main-tls\") pod \"5067c1c4-cfc8-4358-9069-f24576389879\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " Nov 02 07:25:21.323885 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323844 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy\") pod \"5067c1c4-cfc8-4358-9069-f24576389879\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " Nov 02 07:25:21.324027 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323882 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5067c1c4-cfc8-4358-9069-f24576389879\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " Nov 02 07:25:21.324027 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.323972 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-kube-api-access-h6mbb\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-kube-api-access-h6mbb\") pod \"5067c1c4-cfc8-4358-9069-f24576389879\" (UID: \"5067c1c4-cfc8-4358-9069-f24576389879\") " Nov 02 07:25:21.324129 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.324093 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-trusted-ca-bundle" pod "5067c1c4-cfc8-4358-9069-f24576389879" (UID: "5067c1c4-cfc8-4358-9069-f24576389879"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:25:21.324794 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.324294 5899 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:21.324794 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.324321 5899 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-alertmanager-main-db\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:21.324794 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.324568 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-metrics-client-ca" pod "5067c1c4-cfc8-4358-9069-f24576389879" (UID: "5067c1c4-cfc8-4358-9069-f24576389879"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:25:21.326605 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.326576 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-config-out" pod "5067c1c4-cfc8-4358-9069-f24576389879" (UID: "5067c1c4-cfc8-4358-9069-f24576389879"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:25:21.326703 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.326675 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-tls-assets" pod "5067c1c4-cfc8-4358-9069-f24576389879" (UID: "5067c1c4-cfc8-4358-9069-f24576389879"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:25:21.326810 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.326781 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-kube-api-access-h6mbb" pod "5067c1c4-cfc8-4358-9069-f24576389879" (UID: "5067c1c4-cfc8-4358-9069-f24576389879"). InnerVolumeSpecName "kube-api-access-h6mbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:25:21.327738 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.327698 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-config-volume" pod "5067c1c4-cfc8-4358-9069-f24576389879" (UID: "5067c1c4-cfc8-4358-9069-f24576389879"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:21.328342 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.328316 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-web" pod "5067c1c4-cfc8-4358-9069-f24576389879" (UID: "5067c1c4-cfc8-4358-9069-f24576389879"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:21.328643 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.328619 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-metric" pod "5067c1c4-cfc8-4358-9069-f24576389879" (UID: "5067c1c4-cfc8-4358-9069-f24576389879"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:21.328966 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.328947 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy" pod "5067c1c4-cfc8-4358-9069-f24576389879" (UID: "5067c1c4-cfc8-4358-9069-f24576389879"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:21.329043 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.329019 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-main-tls" pod "5067c1c4-cfc8-4358-9069-f24576389879" (UID: "5067c1c4-cfc8-4358-9069-f24576389879"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:21.331758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.331730 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-cluster-tls-config" pod "5067c1c4-cfc8-4358-9069-f24576389879" (UID: "5067c1c4-cfc8-4358-9069-f24576389879"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:21.337827 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.337805 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-web-config" pod "5067c1c4-cfc8-4358-9069-f24576389879" (UID: "5067c1c4-cfc8-4358-9069-f24576389879"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:21.425316 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.425286 5899 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-tls-assets\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:21.425316 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.425314 5899 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-web-config\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:21.425316 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.425323 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5067c1c4-cfc8-4358-9069-f24576389879-metrics-client-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:21.425526 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.425332 5899 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-cluster-tls-config\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:21.425526 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.425342 5899 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-main-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:21.425526 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.425352 5899 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:21.425526 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.425362 5899 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:21.425526 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.425372 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6mbb\" (UniqueName: \"kubernetes.io/projected/5067c1c4-cfc8-4358-9069-f24576389879-kube-api-access-h6mbb\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:21.425526 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.425381 5899 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-config-volume\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:21.425526 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.425389 5899 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5067c1c4-cfc8-4358-9069-f24576389879-config-out\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:21.425526 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.425396 5899 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5067c1c4-cfc8-4358-9069-f24576389879-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:21.972875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.972840 5899 generic.go:356] "Generic (PLEG): container finished" podID="5067c1c4-cfc8-4358-9069-f24576389879" containerID="27128c6aee92197baab9a76c99d441338d6bbdde3c3f2cc99f4fc5493d6f8462" exitCode=0 Nov 02 07:25:21.973073 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.972884 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerDied","Data":"27128c6aee92197baab9a76c99d441338d6bbdde3c3f2cc99f4fc5493d6f8462"} Nov 02 07:25:21.973073 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.972940 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5067c1c4-cfc8-4358-9069-f24576389879","Type":"ContainerDied","Data":"b41ad2171c48e209b8c13466d305258cf3b1051788f44243212abdadebfb1836"} Nov 02 07:25:21.973073 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.972940 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:21.973073 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.972981 5899 scope.go:122] "RemoveContainer" containerID="e7d6f87053a8f2d472dd83cc9d778f0bd1bfb30652c9452f693cbbcfade68c5d" Nov 02 07:25:21.980607 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.980588 5899 scope.go:122] "RemoveContainer" containerID="d108b1ba1274ac67203c71a5c6b5133fdf8402b0707d3229b48bba80d503ddcd" Nov 02 07:25:21.987558 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.987536 5899 scope.go:122] "RemoveContainer" containerID="0f7556c644d1625c07cfb8660f8d413bb604057329cb98f66b9c3f01007c1efc" Nov 02 07:25:21.994429 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:21.994408 5899 scope.go:122] "RemoveContainer" containerID="27128c6aee92197baab9a76c99d441338d6bbdde3c3f2cc99f4fc5493d6f8462" Nov 02 07:25:22.001146 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.001127 5899 scope.go:122] "RemoveContainer" containerID="7dc06196250deaccc91c194e66dc0d821d6d9ec5977a08579c9fae4e5afc2d5a" Nov 02 07:25:22.008889 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.008868 5899 scope.go:122] "RemoveContainer" containerID="f63e67969306d93adfebea905e48f96b10b8be605fa957d8bebded6f85c556d0" Nov 02 07:25:22.016309 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.016287 5899 scope.go:122] "RemoveContainer" containerID="aa3e3d153f3a65cffc261804057dc9dbc532f39e6c0324199097adc4aea6bd2c" Nov 02 07:25:22.023451 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.023432 5899 scope.go:122] "RemoveContainer" containerID="e7d6f87053a8f2d472dd83cc9d778f0bd1bfb30652c9452f693cbbcfade68c5d" Nov 02 07:25:22.023733 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:22.023704 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d6f87053a8f2d472dd83cc9d778f0bd1bfb30652c9452f693cbbcfade68c5d\": container with ID starting with e7d6f87053a8f2d472dd83cc9d778f0bd1bfb30652c9452f693cbbcfade68c5d not found: ID does not exist" containerID="e7d6f87053a8f2d472dd83cc9d778f0bd1bfb30652c9452f693cbbcfade68c5d" Nov 02 07:25:22.023773 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.023743 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d6f87053a8f2d472dd83cc9d778f0bd1bfb30652c9452f693cbbcfade68c5d"} err="failed to get container status \"e7d6f87053a8f2d472dd83cc9d778f0bd1bfb30652c9452f693cbbcfade68c5d\": rpc error: code = NotFound desc = could not find container \"e7d6f87053a8f2d472dd83cc9d778f0bd1bfb30652c9452f693cbbcfade68c5d\": container with ID starting with e7d6f87053a8f2d472dd83cc9d778f0bd1bfb30652c9452f693cbbcfade68c5d not found: ID does not exist" Nov 02 07:25:22.023816 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.023776 5899 scope.go:122] "RemoveContainer" containerID="d108b1ba1274ac67203c71a5c6b5133fdf8402b0707d3229b48bba80d503ddcd" Nov 02 07:25:22.024022 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:22.024006 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d108b1ba1274ac67203c71a5c6b5133fdf8402b0707d3229b48bba80d503ddcd\": container with ID starting with d108b1ba1274ac67203c71a5c6b5133fdf8402b0707d3229b48bba80d503ddcd not found: ID does not exist" containerID="d108b1ba1274ac67203c71a5c6b5133fdf8402b0707d3229b48bba80d503ddcd" Nov 02 07:25:22.024061 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.024032 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d108b1ba1274ac67203c71a5c6b5133fdf8402b0707d3229b48bba80d503ddcd"} err="failed to get container status \"d108b1ba1274ac67203c71a5c6b5133fdf8402b0707d3229b48bba80d503ddcd\": rpc error: code = NotFound desc = could not find container \"d108b1ba1274ac67203c71a5c6b5133fdf8402b0707d3229b48bba80d503ddcd\": container with ID starting with d108b1ba1274ac67203c71a5c6b5133fdf8402b0707d3229b48bba80d503ddcd not found: ID does not exist" Nov 02 07:25:22.024061 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.024054 5899 scope.go:122] "RemoveContainer" containerID="0f7556c644d1625c07cfb8660f8d413bb604057329cb98f66b9c3f01007c1efc" Nov 02 07:25:22.024279 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:22.024260 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f7556c644d1625c07cfb8660f8d413bb604057329cb98f66b9c3f01007c1efc\": container with ID starting with 0f7556c644d1625c07cfb8660f8d413bb604057329cb98f66b9c3f01007c1efc not found: ID does not exist" containerID="0f7556c644d1625c07cfb8660f8d413bb604057329cb98f66b9c3f01007c1efc" Nov 02 07:25:22.024331 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.024284 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7556c644d1625c07cfb8660f8d413bb604057329cb98f66b9c3f01007c1efc"} err="failed to get container status \"0f7556c644d1625c07cfb8660f8d413bb604057329cb98f66b9c3f01007c1efc\": rpc error: code = NotFound desc = could not find container \"0f7556c644d1625c07cfb8660f8d413bb604057329cb98f66b9c3f01007c1efc\": container with ID starting with 0f7556c644d1625c07cfb8660f8d413bb604057329cb98f66b9c3f01007c1efc not found: ID does not exist" Nov 02 07:25:22.024331 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.024297 5899 scope.go:122] "RemoveContainer" containerID="27128c6aee92197baab9a76c99d441338d6bbdde3c3f2cc99f4fc5493d6f8462" Nov 02 07:25:22.024506 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:22.024490 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27128c6aee92197baab9a76c99d441338d6bbdde3c3f2cc99f4fc5493d6f8462\": container with ID starting with 27128c6aee92197baab9a76c99d441338d6bbdde3c3f2cc99f4fc5493d6f8462 not found: ID does not exist" containerID="27128c6aee92197baab9a76c99d441338d6bbdde3c3f2cc99f4fc5493d6f8462" Nov 02 07:25:22.024542 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.024510 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27128c6aee92197baab9a76c99d441338d6bbdde3c3f2cc99f4fc5493d6f8462"} err="failed to get container status \"27128c6aee92197baab9a76c99d441338d6bbdde3c3f2cc99f4fc5493d6f8462\": rpc error: code = NotFound desc = could not find container \"27128c6aee92197baab9a76c99d441338d6bbdde3c3f2cc99f4fc5493d6f8462\": container with ID starting with 27128c6aee92197baab9a76c99d441338d6bbdde3c3f2cc99f4fc5493d6f8462 not found: ID does not exist" Nov 02 07:25:22.024542 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.024523 5899 scope.go:122] "RemoveContainer" containerID="7dc06196250deaccc91c194e66dc0d821d6d9ec5977a08579c9fae4e5afc2d5a" Nov 02 07:25:22.024745 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:22.024728 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc06196250deaccc91c194e66dc0d821d6d9ec5977a08579c9fae4e5afc2d5a\": container with ID starting with 7dc06196250deaccc91c194e66dc0d821d6d9ec5977a08579c9fae4e5afc2d5a not found: ID does not exist" containerID="7dc06196250deaccc91c194e66dc0d821d6d9ec5977a08579c9fae4e5afc2d5a" Nov 02 07:25:22.024781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.024748 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc06196250deaccc91c194e66dc0d821d6d9ec5977a08579c9fae4e5afc2d5a"} err="failed to get container status \"7dc06196250deaccc91c194e66dc0d821d6d9ec5977a08579c9fae4e5afc2d5a\": rpc error: code = NotFound desc = could not find container \"7dc06196250deaccc91c194e66dc0d821d6d9ec5977a08579c9fae4e5afc2d5a\": container with ID starting with 7dc06196250deaccc91c194e66dc0d821d6d9ec5977a08579c9fae4e5afc2d5a not found: ID does not exist" Nov 02 07:25:22.024781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.024760 5899 scope.go:122] "RemoveContainer" containerID="f63e67969306d93adfebea905e48f96b10b8be605fa957d8bebded6f85c556d0" Nov 02 07:25:22.024958 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:22.024943 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63e67969306d93adfebea905e48f96b10b8be605fa957d8bebded6f85c556d0\": container with ID starting with f63e67969306d93adfebea905e48f96b10b8be605fa957d8bebded6f85c556d0 not found: ID does not exist" containerID="f63e67969306d93adfebea905e48f96b10b8be605fa957d8bebded6f85c556d0" Nov 02 07:25:22.025001 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.024961 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63e67969306d93adfebea905e48f96b10b8be605fa957d8bebded6f85c556d0"} err="failed to get container status \"f63e67969306d93adfebea905e48f96b10b8be605fa957d8bebded6f85c556d0\": rpc error: code = NotFound desc = could not find container \"f63e67969306d93adfebea905e48f96b10b8be605fa957d8bebded6f85c556d0\": container with ID starting with f63e67969306d93adfebea905e48f96b10b8be605fa957d8bebded6f85c556d0 not found: ID does not exist" Nov 02 07:25:22.025001 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.024973 5899 scope.go:122] "RemoveContainer" containerID="aa3e3d153f3a65cffc261804057dc9dbc532f39e6c0324199097adc4aea6bd2c" Nov 02 07:25:22.025179 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:22.025147 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3e3d153f3a65cffc261804057dc9dbc532f39e6c0324199097adc4aea6bd2c\": container with ID starting with aa3e3d153f3a65cffc261804057dc9dbc532f39e6c0324199097adc4aea6bd2c not found: ID does not exist" containerID="aa3e3d153f3a65cffc261804057dc9dbc532f39e6c0324199097adc4aea6bd2c" Nov 02 07:25:22.025224 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.025212 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3e3d153f3a65cffc261804057dc9dbc532f39e6c0324199097adc4aea6bd2c"} err="failed to get container status \"aa3e3d153f3a65cffc261804057dc9dbc532f39e6c0324199097adc4aea6bd2c\": rpc error: code = NotFound desc = could not find container \"aa3e3d153f3a65cffc261804057dc9dbc532f39e6c0324199097adc4aea6bd2c\": container with ID starting with aa3e3d153f3a65cffc261804057dc9dbc532f39e6c0324199097adc4aea6bd2c not found: ID does not exist" Nov 02 07:25:22.043215 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.043187 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 07:25:22.056375 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.056346 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 07:25:22.080426 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080348 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 07:25:22.080740 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080719 5899 cpu_manager.go:414] "containerMap: removing container" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="kube-rbac-proxy-metric" Nov 02 07:25:22.080831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080743 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="kube-rbac-proxy-metric" Nov 02 07:25:22.080831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080760 5899 cpu_manager.go:414] "containerMap: removing container" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="alertmanager" Nov 02 07:25:22.080831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080768 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="alertmanager" Nov 02 07:25:22.080831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080780 5899 cpu_manager.go:414] "containerMap: removing container" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="prom-label-proxy" Nov 02 07:25:22.080831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080788 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="prom-label-proxy" Nov 02 07:25:22.080831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080815 5899 cpu_manager.go:414] "containerMap: removing container" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="init-config-reloader" Nov 02 07:25:22.080831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080822 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="init-config-reloader" Nov 02 07:25:22.081196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080839 5899 cpu_manager.go:414] "containerMap: removing container" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="kube-rbac-proxy-web" Nov 02 07:25:22.081196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080848 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="kube-rbac-proxy-web" Nov 02 07:25:22.081196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080868 5899 cpu_manager.go:414] "containerMap: removing container" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="kube-rbac-proxy" Nov 02 07:25:22.081196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080875 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="kube-rbac-proxy" Nov 02 07:25:22.081196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080887 5899 cpu_manager.go:414] "containerMap: removing container" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="config-reloader" Nov 02 07:25:22.081196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080894 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="config-reloader" Nov 02 07:25:22.081196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080966 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="init-config-reloader" Nov 02 07:25:22.081196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080978 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="kube-rbac-proxy-web" Nov 02 07:25:22.081196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080989 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="kube-rbac-proxy" Nov 02 07:25:22.081196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.080998 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="config-reloader" Nov 02 07:25:22.081196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.081007 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="kube-rbac-proxy-metric" Nov 02 07:25:22.081196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.081016 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="alertmanager" Nov 02 07:25:22.081196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.081023 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="5067c1c4-cfc8-4358-9069-f24576389879" containerName="prom-label-proxy" Nov 02 07:25:22.084919 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.084897 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.101414 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.101385 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 07:25:22.101414 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.101386 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 07:25:22.101414 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.101416 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 07:25:22.101603 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.101386 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 07:25:22.101603 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.101473 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 07:25:22.101667 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.101608 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 07:25:22.101667 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.101609 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 07:25:22.101667 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.101658 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-67pc5\"" Nov 02 07:25:22.101762 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.101748 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 07:25:22.115117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.115090 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 07:25:22.116008 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.115988 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 07:25:22.131851 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.131826 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.131995 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.131860 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b36155ab-82c8-456c-8a49-8baaf7006844-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.131995 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.131878 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-config-volume\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.131995 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.131895 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-web-config\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.131995 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.131987 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.132199 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.132078 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.132199 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.132099 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.132301 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.132219 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b36155ab-82c8-456c-8a49-8baaf7006844-config-out\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.132301 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.132266 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b36155ab-82c8-456c-8a49-8baaf7006844-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.132368 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.132315 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.132368 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.132349 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9z8\" (UniqueName: \"kubernetes.io/projected/b36155ab-82c8-456c-8a49-8baaf7006844-kube-api-access-8n9z8\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.132427 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.132396 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b36155ab-82c8-456c-8a49-8baaf7006844-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.132427 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.132423 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b36155ab-82c8-456c-8a49-8baaf7006844-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.232814 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.232772 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b36155ab-82c8-456c-8a49-8baaf7006844-config-out\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.232814 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.232818 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b36155ab-82c8-456c-8a49-8baaf7006844-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.233034 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.232840 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.233034 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.232864 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9z8\" (UniqueName: \"kubernetes.io/projected/b36155ab-82c8-456c-8a49-8baaf7006844-kube-api-access-8n9z8\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.233034 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.232892 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b36155ab-82c8-456c-8a49-8baaf7006844-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.233034 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.232911 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b36155ab-82c8-456c-8a49-8baaf7006844-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.233263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.233027 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.233263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.233078 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b36155ab-82c8-456c-8a49-8baaf7006844-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.233263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.233107 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-config-volume\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.233263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.233128 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-web-config\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.233263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.233154 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.233263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.233232 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.233263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.233238 5899 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 07:25:22.233263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.233257 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.233672 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.233380 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b36155ab-82c8-456c-8a49-8baaf7006844-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.234137 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.233880 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b36155ab-82c8-456c-8a49-8baaf7006844-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.234137 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.233940 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b36155ab-82c8-456c-8a49-8baaf7006844-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.235945 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.235842 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b36155ab-82c8-456c-8a49-8baaf7006844-config-out\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.236335 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.236267 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.236335 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.236277 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b36155ab-82c8-456c-8a49-8baaf7006844-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.236638 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.236618 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.236731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.236656 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-web-config\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.236731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.236696 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.236926 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.236907 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-config-volume\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.237422 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.237403 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.237716 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.237694 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b36155ab-82c8-456c-8a49-8baaf7006844-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.257866 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.257840 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9z8\" (UniqueName: \"kubernetes.io/projected/b36155ab-82c8-456c-8a49-8baaf7006844-kube-api-access-8n9z8\") pod \"alertmanager-main-0\" (UID: \"b36155ab-82c8-456c-8a49-8baaf7006844\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.404330 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.404241 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 07:25:22.733691 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.733663 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 07:25:22.735316 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:25:22.735284 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36155ab_82c8_456c_8a49_8baaf7006844.slice/crio-b98e02e5b767d2ea1bb13a8f91fa12f57e693dfa20df5f7c96dd6124a9862cf4 WatchSource:0}: Error finding container b98e02e5b767d2ea1bb13a8f91fa12f57e693dfa20df5f7c96dd6124a9862cf4: Status 404 returned error can't find the container with id b98e02e5b767d2ea1bb13a8f91fa12f57e693dfa20df5f7c96dd6124a9862cf4 Nov 02 07:25:22.978539 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.978507 5899 generic.go:356] "Generic (PLEG): container finished" podID="b36155ab-82c8-456c-8a49-8baaf7006844" containerID="ce39f0cf439ed77fddcb5cb2903ba2bf378c94bedc01fae8facc135f57042231" exitCode=0 Nov 02 07:25:22.978539 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.978542 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b36155ab-82c8-456c-8a49-8baaf7006844","Type":"ContainerDied","Data":"ce39f0cf439ed77fddcb5cb2903ba2bf378c94bedc01fae8facc135f57042231"} Nov 02 07:25:22.978824 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:22.978594 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b36155ab-82c8-456c-8a49-8baaf7006844","Type":"ContainerStarted","Data":"b98e02e5b767d2ea1bb13a8f91fa12f57e693dfa20df5f7c96dd6124a9862cf4"} Nov 02 07:25:23.817728 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:23.817695 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5067c1c4-cfc8-4358-9069-f24576389879" path="/var/lib/kubelet/pods/5067c1c4-cfc8-4358-9069-f24576389879/volumes" Nov 02 07:25:23.985523 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:23.985482 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b36155ab-82c8-456c-8a49-8baaf7006844","Type":"ContainerStarted","Data":"e271f416716365fd9ba7b6a80db9d15e0ada3917845db8ee44a0605f06041057"} Nov 02 07:25:23.985682 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:23.985568 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b36155ab-82c8-456c-8a49-8baaf7006844","Type":"ContainerStarted","Data":"8dbeb3e451e8bf64439e35113fa8855c6ce3799dcbea31b58567ce3a5148a991"} Nov 02 07:25:23.985682 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:23.985608 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b36155ab-82c8-456c-8a49-8baaf7006844","Type":"ContainerStarted","Data":"ae7287a7bd3d1a6123b225672841babbd266cbb69aec7e62a4fb41fa46130106"} Nov 02 07:25:23.985682 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:23.985645 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b36155ab-82c8-456c-8a49-8baaf7006844","Type":"ContainerStarted","Data":"fabe4a335ffe6fda7a954c8739d978a2d27369e4efb8ad2b69b7e45594e7fc04"} Nov 02 07:25:23.985788 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:23.985683 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b36155ab-82c8-456c-8a49-8baaf7006844","Type":"ContainerStarted","Data":"eef9f57722c89829484fcbb6832cbf4202110ec94f25f127cb0957dfda23c157"} Nov 02 07:25:23.985788 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:23.985715 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b36155ab-82c8-456c-8a49-8baaf7006844","Type":"ContainerStarted","Data":"ca4a066058f4ea5ad8167bf507b0ab21ae54906418bc9cd99043c351c0881a45"} Nov 02 07:25:24.021935 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.021903 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7946767c4b-g78ln"] Nov 02 07:25:24.024451 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.024433 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.044474 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.044445 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 07:25:24.044474 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.044443 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 07:25:24.044610 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.044475 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 07:25:24.044610 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.044475 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 07:25:24.044610 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.044517 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-znj5s\"" Nov 02 07:25:24.048277 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.048255 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.048447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.048429 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-secret-telemeter-client\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.048520 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.048454 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2d5g\" (UniqueName: \"kubernetes.io/projected/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-kube-api-access-n2d5g\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.048520 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.048496 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.048701 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.048681 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-federate-client-tls\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.048756 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.048715 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-metrics-client-ca\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.048832 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.048809 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-telemeter-client-tls\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.048906 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.048890 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-serving-certs-ca-bundle\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.058193 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.058108 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.044711867 podStartE2EDuration="2.058096172s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.384305ms" isStatefulPod=false podCreationTimestamp="2026-11-02 07:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:25:24.044796113 +0000 UTC m=+24.343076938" watchObservedRunningTime="2026-11-02 07:25:24.058096172 +0000 UTC m=+24.356376997" Nov 02 07:25:24.099530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.099448 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7946767c4b-g78ln"] Nov 02 07:25:24.125229 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.125189 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 07:25:24.138708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.138685 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 07:25:24.149928 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.149902 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-federate-client-tls\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.150042 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.149935 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-metrics-client-ca\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.150042 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.149957 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-telemeter-client-tls\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.150042 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.149985 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-serving-certs-ca-bundle\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.150148 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.150110 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.150218 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.150184 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-secret-telemeter-client\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.150218 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.150207 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2d5g\" (UniqueName: \"kubernetes.io/projected/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-kube-api-access-n2d5g\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.150326 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.150239 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.150825 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.150801 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-serving-certs-ca-bundle\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.150931 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.150818 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-metrics-client-ca\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.151300 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.151280 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.152454 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.152434 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-federate-client-tls\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.152792 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.152776 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-secret-telemeter-client\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.152928 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.152907 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.152971 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.152958 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-telemeter-client-tls\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.174091 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.174065 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2d5g\" (UniqueName: \"kubernetes.io/projected/f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8-kube-api-access-n2d5g\") pod \"telemeter-client-7946767c4b-g78ln\" (UID: \"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8\") " pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.337273 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.337235 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" Nov 02 07:25:24.658748 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.658721 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7946767c4b-g78ln"] Nov 02 07:25:24.660787 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:25:24.660759 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3301cf0_cecf_4a6a_a1b7_1be6d87b61c8.slice/crio-1fbc1cd87eabbdf3686e982b365a880c367bb18e5083a53ccc241514b24073cd WatchSource:0}: Error finding container 1fbc1cd87eabbdf3686e982b365a880c367bb18e5083a53ccc241514b24073cd: Status 404 returned error can't find the container with id 1fbc1cd87eabbdf3686e982b365a880c367bb18e5083a53ccc241514b24073cd Nov 02 07:25:24.662957 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.662942 5899 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 07:25:24.689766 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.689738 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 07:25:24.690053 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.690020 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="prometheus" containerID="cri-o://a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3" gracePeriod=600 Nov 02 07:25:24.690208 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.690051 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="config-reloader" containerID="cri-o://3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc" gracePeriod=600 Nov 02 07:25:24.690208 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.690056 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="thanos-sidecar" containerID="cri-o://93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62" gracePeriod=600 Nov 02 07:25:24.690208 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.690029 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="kube-rbac-proxy" containerID="cri-o://e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c" gracePeriod=600 Nov 02 07:25:24.690208 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.690056 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd" gracePeriod=600 Nov 02 07:25:24.690208 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.690062 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="kube-rbac-proxy-web" containerID="cri-o://311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5" gracePeriod=600 Nov 02 07:25:24.929813 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.929789 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:24.957665 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.957631 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-grpc-tls\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.957819 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.957696 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-web-config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-web-config\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.957819 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.957795 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-tls\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.957915 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.957833 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-kubelet-serving-ca-bundle\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.957915 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.957869 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-0\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958025 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.957992 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config-out\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958080 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958040 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-trusted-ca-bundle\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958136 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958074 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-serving-certs-ca-bundle\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958136 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958115 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958261 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958144 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958261 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958223 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-thanos-prometheus-http-client-file\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958261 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958253 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-kube-api-access-n4hv6\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-kube-api-access-n4hv6\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958414 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958260 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-kubelet-serving-ca-bundle" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:25:24.958414 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958283 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-kube-rbac-proxy\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958624 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-tls-assets\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-tls-assets\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958681 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-2\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958721 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-metrics-client-certs\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958839 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958758 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-db\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958839 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958790 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-metrics-client-ca\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958937 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958901 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.958988 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.958937 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-1\") pod \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\" (UID: \"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8\") " Nov 02 07:25:24.959177 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.959128 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-trusted-ca-bundle" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:25:24.959313 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.959293 5899 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:24.959437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.959420 5899 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:24.959997 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.959521 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-serving-certs-ca-bundle" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:25:24.959997 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.959895 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-1" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:25:24.961077 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.960606 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-metrics-client-ca" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:25:24.961077 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.960646 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-grpc-tls" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:24.961077 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.960914 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-0" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:25:24.961077 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.961041 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-2" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:25:24.961846 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.961805 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-db" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:25:24.963384 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.963356 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-kube-rbac-proxy" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:24.963553 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.963524 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-metrics-client-certs" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:24.963688 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.963648 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-tls" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:24.963851 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.963778 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-thanos-sidecar-tls" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:24.963851 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.963811 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config-out" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:25:24.963851 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.963825 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-kube-rbac-proxy-web" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:24.963978 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.963928 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-thanos-prometheus-http-client-file" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:24.964345 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.964325 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-tls-assets" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:25:24.964701 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.964681 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-kube-api-access-n4hv6" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "kube-api-access-n4hv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:25:24.964777 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.964753 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:24.974122 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.974095 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-web-config" pod "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" (UID: "63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:25:24.992232 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.992202 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" event={"ID":"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8","Type":"ContainerStarted","Data":"1fbc1cd87eabbdf3686e982b365a880c367bb18e5083a53ccc241514b24073cd"} Nov 02 07:25:24.996017 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.995985 5899 generic.go:356] "Generic (PLEG): container finished" podID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerID="6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd" exitCode=0 Nov 02 07:25:24.996017 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996006 5899 generic.go:356] "Generic (PLEG): container finished" podID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerID="e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c" exitCode=0 Nov 02 07:25:24.996017 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996012 5899 generic.go:356] "Generic (PLEG): container finished" podID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerID="311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5" exitCode=0 Nov 02 07:25:24.996017 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996019 5899 generic.go:356] "Generic (PLEG): container finished" podID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerID="93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62" exitCode=0 Nov 02 07:25:24.996017 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996024 5899 generic.go:356] "Generic (PLEG): container finished" podID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerID="3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc" exitCode=0 Nov 02 07:25:24.996313 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996029 5899 generic.go:356] "Generic (PLEG): container finished" podID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerID="a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3" exitCode=0 Nov 02 07:25:24.996313 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996066 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerDied","Data":"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd"} Nov 02 07:25:24.996313 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996103 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:24.996313 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996133 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerDied","Data":"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c"} Nov 02 07:25:24.996313 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996178 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerDied","Data":"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5"} Nov 02 07:25:24.996313 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996190 5899 scope.go:122] "RemoveContainer" containerID="6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd" Nov 02 07:25:24.996313 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996207 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerDied","Data":"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62"} Nov 02 07:25:24.996313 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996231 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerDied","Data":"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc"} Nov 02 07:25:24.996313 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996256 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerDied","Data":"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3"} Nov 02 07:25:24.996313 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:24.996280 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8","Type":"ContainerDied","Data":"92d751fe6bb0ba81d295c92ae8ca43233805fd440339cad982dd6fc8c293ad43"} Nov 02 07:25:25.006920 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.006770 5899 scope.go:122] "RemoveContainer" containerID="e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c" Nov 02 07:25:25.016100 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.016072 5899 scope.go:122] "RemoveContainer" containerID="311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5" Nov 02 07:25:25.023849 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.023826 5899 scope.go:122] "RemoveContainer" containerID="93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62" Nov 02 07:25:25.030774 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.030757 5899 scope.go:122] "RemoveContainer" containerID="3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc" Nov 02 07:25:25.037869 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.037842 5899 scope.go:122] "RemoveContainer" containerID="a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3" Nov 02 07:25:25.049113 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.049095 5899 scope.go:122] "RemoveContainer" containerID="4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a" Nov 02 07:25:25.055782 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.055760 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 07:25:25.056523 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.056507 5899 scope.go:122] "RemoveContainer" containerID="6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd" Nov 02 07:25:25.056804 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:25.056777 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd\": container with ID starting with 6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd not found: ID does not exist" containerID="6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd" Nov 02 07:25:25.056878 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.056809 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd"} err="failed to get container status \"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd\": rpc error: code = NotFound desc = could not find container \"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd\": container with ID starting with 6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd not found: ID does not exist" Nov 02 07:25:25.056878 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.056835 5899 scope.go:122] "RemoveContainer" containerID="e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c" Nov 02 07:25:25.057130 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:25.057114 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c\": container with ID starting with e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c not found: ID does not exist" containerID="e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c" Nov 02 07:25:25.057302 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.057136 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c"} err="failed to get container status \"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c\": rpc error: code = NotFound desc = could not find container \"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c\": container with ID starting with e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c not found: ID does not exist" Nov 02 07:25:25.057302 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.057156 5899 scope.go:122] "RemoveContainer" containerID="311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5" Nov 02 07:25:25.057465 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:25.057441 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5\": container with ID starting with 311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5 not found: ID does not exist" containerID="311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5" Nov 02 07:25:25.057513 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.057474 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5"} err="failed to get container status \"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5\": rpc error: code = NotFound desc = could not find container \"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5\": container with ID starting with 311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5 not found: ID does not exist" Nov 02 07:25:25.057513 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.057494 5899 scope.go:122] "RemoveContainer" containerID="93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62" Nov 02 07:25:25.057751 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:25.057731 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62\": container with ID starting with 93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62 not found: ID does not exist" containerID="93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62" Nov 02 07:25:25.057800 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.057756 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62"} err="failed to get container status \"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62\": rpc error: code = NotFound desc = could not find container \"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62\": container with ID starting with 93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62 not found: ID does not exist" Nov 02 07:25:25.057800 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.057770 5899 scope.go:122] "RemoveContainer" containerID="3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc" Nov 02 07:25:25.057981 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:25.057965 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc\": container with ID starting with 3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc not found: ID does not exist" containerID="3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc" Nov 02 07:25:25.058021 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.057984 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc"} err="failed to get container status \"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc\": rpc error: code = NotFound desc = could not find container \"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc\": container with ID starting with 3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc not found: ID does not exist" Nov 02 07:25:25.058021 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.057996 5899 scope.go:122] "RemoveContainer" containerID="a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3" Nov 02 07:25:25.058529 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:25.058511 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3\": container with ID starting with a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3 not found: ID does not exist" containerID="a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3" Nov 02 07:25:25.058576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.058532 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3"} err="failed to get container status \"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3\": rpc error: code = NotFound desc = could not find container \"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3\": container with ID starting with a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3 not found: ID does not exist" Nov 02 07:25:25.058576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.058548 5899 scope.go:122] "RemoveContainer" containerID="4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a" Nov 02 07:25:25.058800 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:25.058778 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a\": container with ID starting with 4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a not found: ID does not exist" containerID="4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a" Nov 02 07:25:25.058851 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.058809 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a"} err="failed to get container status \"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a\": rpc error: code = NotFound desc = could not find container \"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a\": container with ID starting with 4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a not found: ID does not exist" Nov 02 07:25:25.058851 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.058832 5899 scope.go:122] "RemoveContainer" containerID="6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd" Nov 02 07:25:25.059070 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.059048 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd"} err="failed to get container status \"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd\": rpc error: code = NotFound desc = could not find container \"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd\": container with ID starting with 6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd not found: ID does not exist" Nov 02 07:25:25.059143 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.059071 5899 scope.go:122] "RemoveContainer" containerID="e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c" Nov 02 07:25:25.059320 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.059301 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c"} err="failed to get container status \"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c\": rpc error: code = NotFound desc = could not find container \"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c\": container with ID starting with e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c not found: ID does not exist" Nov 02 07:25:25.059320 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.059322 5899 scope.go:122] "RemoveContainer" containerID="311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5" Nov 02 07:25:25.059552 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.059536 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5"} err="failed to get container status \"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5\": rpc error: code = NotFound desc = could not find container \"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5\": container with ID starting with 311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5 not found: ID does not exist" Nov 02 07:25:25.059631 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.059553 5899 scope.go:122] "RemoveContainer" containerID="93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62" Nov 02 07:25:25.059828 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.059799 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62"} err="failed to get container status \"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62\": rpc error: code = NotFound desc = could not find container \"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62\": container with ID starting with 93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62 not found: ID does not exist" Nov 02 07:25:25.059828 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.059825 5899 scope.go:122] "RemoveContainer" containerID="3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc" Nov 02 07:25:25.060139 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060122 5899 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-web-config\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060379 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060143 5899 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060379 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060190 5899 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060379 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060209 5899 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config-out\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060379 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060222 5899 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060379 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060237 5899 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060379 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060262 5899 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060379 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060278 5899 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060379 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060292 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4hv6\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-kube-api-access-n4hv6\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060379 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060307 5899 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-kube-rbac-proxy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060818 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060480 5899 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-tls-assets\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060818 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060502 5899 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060818 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060516 5899 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-metrics-client-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060818 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060533 5899 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-db\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060818 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060557 5899 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-configmap-metrics-client-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060818 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060571 5899 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-config\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060818 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060585 5899 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.060818 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060599 5899 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8-secret-grpc-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:25:25.061116 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060903 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc"} err="failed to get container status \"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc\": rpc error: code = NotFound desc = could not find container \"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc\": container with ID starting with 3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc not found: ID does not exist" Nov 02 07:25:25.061116 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.060934 5899 scope.go:122] "RemoveContainer" containerID="a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3" Nov 02 07:25:25.061231 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.061208 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3"} err="failed to get container status \"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3\": rpc error: code = NotFound desc = could not find container \"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3\": container with ID starting with a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3 not found: ID does not exist" Nov 02 07:25:25.061274 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.061234 5899 scope.go:122] "RemoveContainer" containerID="4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a" Nov 02 07:25:25.061496 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.061474 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a"} err="failed to get container status \"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a\": rpc error: code = NotFound desc = could not find container \"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a\": container with ID starting with 4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a not found: ID does not exist" Nov 02 07:25:25.061553 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.061497 5899 scope.go:122] "RemoveContainer" containerID="6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd" Nov 02 07:25:25.061742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.061722 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd"} err="failed to get container status \"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd\": rpc error: code = NotFound desc = could not find container \"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd\": container with ID starting with 6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd not found: ID does not exist" Nov 02 07:25:25.061742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.061741 5899 scope.go:122] "RemoveContainer" containerID="e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c" Nov 02 07:25:25.062006 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.061986 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c"} err="failed to get container status \"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c\": rpc error: code = NotFound desc = could not find container \"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c\": container with ID starting with e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c not found: ID does not exist" Nov 02 07:25:25.062006 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.062005 5899 scope.go:122] "RemoveContainer" containerID="311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5" Nov 02 07:25:25.062339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.062317 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5"} err="failed to get container status \"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5\": rpc error: code = NotFound desc = could not find container \"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5\": container with ID starting with 311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5 not found: ID does not exist" Nov 02 07:25:25.062411 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.062340 5899 scope.go:122] "RemoveContainer" containerID="93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62" Nov 02 07:25:25.062679 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.062652 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62"} err="failed to get container status \"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62\": rpc error: code = NotFound desc = could not find container \"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62\": container with ID starting with 93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62 not found: ID does not exist" Nov 02 07:25:25.062776 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.062679 5899 scope.go:122] "RemoveContainer" containerID="3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc" Nov 02 07:25:25.062916 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.062898 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 07:25:25.063008 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.062975 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc"} err="failed to get container status \"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc\": rpc error: code = NotFound desc = could not find container \"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc\": container with ID starting with 3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc not found: ID does not exist" Nov 02 07:25:25.063085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.062999 5899 scope.go:122] "RemoveContainer" containerID="a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3" Nov 02 07:25:25.063263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.063244 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3"} err="failed to get container status \"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3\": rpc error: code = NotFound desc = could not find container \"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3\": container with ID starting with a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3 not found: ID does not exist" Nov 02 07:25:25.063338 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.063264 5899 scope.go:122] "RemoveContainer" containerID="4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a" Nov 02 07:25:25.063533 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.063483 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a"} err="failed to get container status \"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a\": rpc error: code = NotFound desc = could not find container \"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a\": container with ID starting with 4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a not found: ID does not exist" Nov 02 07:25:25.063569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.063535 5899 scope.go:122] "RemoveContainer" containerID="6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd" Nov 02 07:25:25.063836 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.063806 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd"} err="failed to get container status \"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd\": rpc error: code = NotFound desc = could not find container \"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd\": container with ID starting with 6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd not found: ID does not exist" Nov 02 07:25:25.063836 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.063825 5899 scope.go:122] "RemoveContainer" containerID="e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c" Nov 02 07:25:25.064063 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.064041 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c"} err="failed to get container status \"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c\": rpc error: code = NotFound desc = could not find container \"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c\": container with ID starting with e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c not found: ID does not exist" Nov 02 07:25:25.064130 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.064063 5899 scope.go:122] "RemoveContainer" containerID="311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5" Nov 02 07:25:25.064307 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.064289 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5"} err="failed to get container status \"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5\": rpc error: code = NotFound desc = could not find container \"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5\": container with ID starting with 311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5 not found: ID does not exist" Nov 02 07:25:25.064371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.064309 5899 scope.go:122] "RemoveContainer" containerID="93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62" Nov 02 07:25:25.064530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.064514 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62"} err="failed to get container status \"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62\": rpc error: code = NotFound desc = could not find container \"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62\": container with ID starting with 93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62 not found: ID does not exist" Nov 02 07:25:25.064618 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.064532 5899 scope.go:122] "RemoveContainer" containerID="3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc" Nov 02 07:25:25.064761 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.064745 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc"} err="failed to get container status \"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc\": rpc error: code = NotFound desc = could not find container \"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc\": container with ID starting with 3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc not found: ID does not exist" Nov 02 07:25:25.064831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.064763 5899 scope.go:122] "RemoveContainer" containerID="a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3" Nov 02 07:25:25.064979 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.064963 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3"} err="failed to get container status \"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3\": rpc error: code = NotFound desc = could not find container \"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3\": container with ID starting with a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3 not found: ID does not exist" Nov 02 07:25:25.064979 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.064979 5899 scope.go:122] "RemoveContainer" containerID="4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a" Nov 02 07:25:25.065212 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.065196 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a"} err="failed to get container status \"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a\": rpc error: code = NotFound desc = could not find container \"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a\": container with ID starting with 4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a not found: ID does not exist" Nov 02 07:25:25.065259 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.065213 5899 scope.go:122] "RemoveContainer" containerID="6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd" Nov 02 07:25:25.065419 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.065402 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd"} err="failed to get container status \"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd\": rpc error: code = NotFound desc = could not find container \"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd\": container with ID starting with 6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd not found: ID does not exist" Nov 02 07:25:25.065464 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.065418 5899 scope.go:122] "RemoveContainer" containerID="e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c" Nov 02 07:25:25.065623 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.065606 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c"} err="failed to get container status \"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c\": rpc error: code = NotFound desc = could not find container \"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c\": container with ID starting with e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c not found: ID does not exist" Nov 02 07:25:25.065683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.065624 5899 scope.go:122] "RemoveContainer" containerID="311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5" Nov 02 07:25:25.065840 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.065825 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5"} err="failed to get container status \"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5\": rpc error: code = NotFound desc = could not find container \"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5\": container with ID starting with 311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5 not found: ID does not exist" Nov 02 07:25:25.065884 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.065839 5899 scope.go:122] "RemoveContainer" containerID="93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62" Nov 02 07:25:25.066039 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.066022 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62"} err="failed to get container status \"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62\": rpc error: code = NotFound desc = could not find container \"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62\": container with ID starting with 93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62 not found: ID does not exist" Nov 02 07:25:25.066100 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.066039 5899 scope.go:122] "RemoveContainer" containerID="3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc" Nov 02 07:25:25.066267 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.066249 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc"} err="failed to get container status \"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc\": rpc error: code = NotFound desc = could not find container \"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc\": container with ID starting with 3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc not found: ID does not exist" Nov 02 07:25:25.066320 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.066267 5899 scope.go:122] "RemoveContainer" containerID="a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3" Nov 02 07:25:25.066440 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.066422 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3"} err="failed to get container status \"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3\": rpc error: code = NotFound desc = could not find container \"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3\": container with ID starting with a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3 not found: ID does not exist" Nov 02 07:25:25.066502 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.066441 5899 scope.go:122] "RemoveContainer" containerID="4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a" Nov 02 07:25:25.066668 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.066654 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a"} err="failed to get container status \"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a\": rpc error: code = NotFound desc = could not find container \"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a\": container with ID starting with 4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a not found: ID does not exist" Nov 02 07:25:25.066722 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.066669 5899 scope.go:122] "RemoveContainer" containerID="6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd" Nov 02 07:25:25.066904 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.066886 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd"} err="failed to get container status \"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd\": rpc error: code = NotFound desc = could not find container \"6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd\": container with ID starting with 6c4993058d25193e1580894769d7b470108f8d0ef37a642c6a0f3395f3aa6ecd not found: ID does not exist" Nov 02 07:25:25.066954 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.066905 5899 scope.go:122] "RemoveContainer" containerID="e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c" Nov 02 07:25:25.067092 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.067077 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c"} err="failed to get container status \"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c\": rpc error: code = NotFound desc = could not find container \"e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c\": container with ID starting with e65dd527aa2c324e4977dfca0b2cb072b0ba8e432e9b8b5f03327aefadd6661c not found: ID does not exist" Nov 02 07:25:25.067139 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.067092 5899 scope.go:122] "RemoveContainer" containerID="311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5" Nov 02 07:25:25.067291 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.067270 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5"} err="failed to get container status \"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5\": rpc error: code = NotFound desc = could not find container \"311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5\": container with ID starting with 311e262ff0759ac0c71648638b883000f892e3fb0449f4be340f29636b8250c5 not found: ID does not exist" Nov 02 07:25:25.067338 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.067290 5899 scope.go:122] "RemoveContainer" containerID="93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62" Nov 02 07:25:25.067544 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.067526 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62"} err="failed to get container status \"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62\": rpc error: code = NotFound desc = could not find container \"93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62\": container with ID starting with 93fec9eeeb46d8eeaae250fa1a50eadb65f3f2250c0fd2ed9cde452d6ef96a62 not found: ID does not exist" Nov 02 07:25:25.067622 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.067545 5899 scope.go:122] "RemoveContainer" containerID="3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc" Nov 02 07:25:25.067785 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.067770 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc"} err="failed to get container status \"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc\": rpc error: code = NotFound desc = could not find container \"3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc\": container with ID starting with 3b230ae3cbe07297c28cc6416520ceb08e9b69a505e96a6e6243dd2333891dcc not found: ID does not exist" Nov 02 07:25:25.067851 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.067786 5899 scope.go:122] "RemoveContainer" containerID="a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3" Nov 02 07:25:25.068059 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.068041 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3"} err="failed to get container status \"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3\": rpc error: code = NotFound desc = could not find container \"a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3\": container with ID starting with a106b2f058b07894e6c4fc7a0a3dbcbe4eb9298d83ed179569d65d8c1811e4e3 not found: ID does not exist" Nov 02 07:25:25.068102 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.068059 5899 scope.go:122] "RemoveContainer" containerID="4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a" Nov 02 07:25:25.068288 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.068267 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a"} err="failed to get container status \"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a\": rpc error: code = NotFound desc = could not find container \"4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a\": container with ID starting with 4274e1c845bf405417edc8ee8547d3f81696b220fb7b519b6d757496262b641a not found: ID does not exist" Nov 02 07:25:25.096055 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096023 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 07:25:25.096529 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096501 5899 cpu_manager.go:414] "containerMap: removing container" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="thanos-sidecar" Nov 02 07:25:25.096529 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096522 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="thanos-sidecar" Nov 02 07:25:25.096663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096539 5899 cpu_manager.go:414] "containerMap: removing container" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="prometheus" Nov 02 07:25:25.096663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096548 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="prometheus" Nov 02 07:25:25.096663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096567 5899 cpu_manager.go:414] "containerMap: removing container" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="kube-rbac-proxy-thanos" Nov 02 07:25:25.096663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096574 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="kube-rbac-proxy-thanos" Nov 02 07:25:25.096663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096588 5899 cpu_manager.go:414] "containerMap: removing container" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="kube-rbac-proxy-web" Nov 02 07:25:25.096663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096597 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="kube-rbac-proxy-web" Nov 02 07:25:25.096663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096615 5899 cpu_manager.go:414] "containerMap: removing container" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="init-config-reloader" Nov 02 07:25:25.096663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096623 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="init-config-reloader" Nov 02 07:25:25.096663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096637 5899 cpu_manager.go:414] "containerMap: removing container" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="kube-rbac-proxy" Nov 02 07:25:25.096663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096644 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="kube-rbac-proxy" Nov 02 07:25:25.096663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096654 5899 cpu_manager.go:414] "containerMap: removing container" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="config-reloader" Nov 02 07:25:25.096663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096661 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="config-reloader" Nov 02 07:25:25.097097 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096722 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="init-config-reloader" Nov 02 07:25:25.097097 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096736 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="kube-rbac-proxy" Nov 02 07:25:25.097097 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096745 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="config-reloader" Nov 02 07:25:25.097097 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096756 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="thanos-sidecar" Nov 02 07:25:25.097097 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096765 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="prometheus" Nov 02 07:25:25.097097 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096775 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="kube-rbac-proxy-thanos" Nov 02 07:25:25.097097 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.096784 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" containerName="kube-rbac-proxy-web" Nov 02 07:25:25.099561 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.099534 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.116271 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.116242 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 07:25:25.116271 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.116253 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 07:25:25.116440 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.116253 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 07:25:25.116440 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.116345 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 07:25:25.116710 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.116626 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 07:25:25.116710 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.116680 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 07:25:25.129876 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.129824 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 07:25:25.129876 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.129824 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 07:25:25.130131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.129984 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 07:25:25.130131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.129980 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 07:25:25.130131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.130037 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-32bjlgfpvos9j\"" Nov 02 07:25:25.130131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.130049 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 07:25:25.130131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.130062 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 07:25:25.130131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.130109 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-45mkt\"" Nov 02 07:25:25.130992 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.130968 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 07:25:25.161374 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.161340 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4602a0a9-fa2d-4233-b04b-0a33011f8021-config-out\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.161561 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.161382 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.161561 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.161402 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.161561 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.161426 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.161561 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.161524 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.161734 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.161566 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.161734 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.161683 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.161734 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.161714 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.161848 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.161739 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.161848 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.161801 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.161920 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.161867 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-config\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.161920 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.161910 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.161981 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.161964 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.162027 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.162011 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7jjb\" (UniqueName: \"kubernetes.io/projected/4602a0a9-fa2d-4233-b04b-0a33011f8021-kube-api-access-j7jjb\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.162085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.162043 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.162138 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.162095 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.162138 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.162125 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.162264 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.162154 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.162264 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.162232 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-web-config\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.162366 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.162314 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4602a0a9-fa2d-4233-b04b-0a33011f8021-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.216955 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.216917 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 07:25:25.220184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.218420 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 07:25:25.263290 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263245 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7jjb\" (UniqueName: \"kubernetes.io/projected/4602a0a9-fa2d-4233-b04b-0a33011f8021-kube-api-access-j7jjb\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263470 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263308 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263470 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263340 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263470 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263374 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263470 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263404 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263470 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263438 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-web-config\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263472 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4602a0a9-fa2d-4233-b04b-0a33011f8021-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263503 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4602a0a9-fa2d-4233-b04b-0a33011f8021-config-out\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263537 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263558 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263580 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263607 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263629 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263680 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263706 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.263742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263730 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.264252 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263751 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.264252 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263784 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-config\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.264252 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263816 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.264252 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.263840 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.264252 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.264125 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.264495 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.264247 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.264495 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.264484 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.267016 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.265374 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.267016 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.265411 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.267016 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.265887 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.267016 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.266525 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4602a0a9-fa2d-4233-b04b-0a33011f8021-config-out\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.267016 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.266939 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-web-config\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.267373 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.267329 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4602a0a9-fa2d-4233-b04b-0a33011f8021-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.267482 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.267458 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.268298 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.268245 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.268791 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.268767 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.269404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.269346 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.269620 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.269568 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.269953 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.269923 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4602a0a9-fa2d-4233-b04b-0a33011f8021-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.270249 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.270229 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-config\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.270904 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.270881 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.271129 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.271107 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.272243 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.272225 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4602a0a9-fa2d-4233-b04b-0a33011f8021-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.287891 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.287866 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7jjb\" (UniqueName: \"kubernetes.io/projected/4602a0a9-fa2d-4233-b04b-0a33011f8021-kube-api-access-j7jjb\") pod \"prometheus-k8s-0\" (UID: \"4602a0a9-fa2d-4233-b04b-0a33011f8021\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.422274 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.422236 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:25.755743 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.755720 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 07:25:25.757598 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:25:25.757575 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4602a0a9_fa2d_4233_b04b_0a33011f8021.slice/crio-5bc8e288419d952d77a450e12e5e16ee05976d39a26a25b75e7224ea3c53b56f WatchSource:0}: Error finding container 5bc8e288419d952d77a450e12e5e16ee05976d39a26a25b75e7224ea3c53b56f: Status 404 returned error can't find the container with id 5bc8e288419d952d77a450e12e5e16ee05976d39a26a25b75e7224ea3c53b56f Nov 02 07:25:25.819478 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:25.819442 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8" path="/var/lib/kubelet/pods/63e5522b-3924-4cc4-8e6a-34e7e7c2e0e8/volumes" Nov 02 07:25:26.003246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:26.003208 5899 generic.go:356] "Generic (PLEG): container finished" podID="4602a0a9-fa2d-4233-b04b-0a33011f8021" containerID="720a74d2b6f1f2d04fb4628a9723dd195bfc5535beb40851b4e44ce0dc42ed80" exitCode=0 Nov 02 07:25:26.003729 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:26.003298 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4602a0a9-fa2d-4233-b04b-0a33011f8021","Type":"ContainerDied","Data":"720a74d2b6f1f2d04fb4628a9723dd195bfc5535beb40851b4e44ce0dc42ed80"} Nov 02 07:25:26.003729 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:26.003391 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4602a0a9-fa2d-4233-b04b-0a33011f8021","Type":"ContainerStarted","Data":"5bc8e288419d952d77a450e12e5e16ee05976d39a26a25b75e7224ea3c53b56f"} Nov 02 07:25:27.010562 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.010525 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4602a0a9-fa2d-4233-b04b-0a33011f8021","Type":"ContainerStarted","Data":"728f8b5818c162cc1e468ba2999dcc3e3b60ef4c944569b22a17a9b4420df6c0"} Nov 02 07:25:27.011031 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.010613 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4602a0a9-fa2d-4233-b04b-0a33011f8021","Type":"ContainerStarted","Data":"5dc460cae8f97a72796953a5e79ad99a3a38a975ca29f51d3645c42b6b634185"} Nov 02 07:25:27.011031 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.010647 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4602a0a9-fa2d-4233-b04b-0a33011f8021","Type":"ContainerStarted","Data":"00ac472cec6a1b573f560028fa0a6721a4373f84ddf5c8bb6d77425b4687c2fa"} Nov 02 07:25:27.011031 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.010670 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4602a0a9-fa2d-4233-b04b-0a33011f8021","Type":"ContainerStarted","Data":"9e0563e1ff959a8c5e87b1f7af9247a81fb24f013c772885588fb10bda6d0dd4"} Nov 02 07:25:27.011031 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.010703 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4602a0a9-fa2d-4233-b04b-0a33011f8021","Type":"ContainerStarted","Data":"d87330ae54cfb726d166bc8cd85a7176ab3b870d61269f9f246a6fa823329595"} Nov 02 07:25:27.011031 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.010741 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4602a0a9-fa2d-4233-b04b-0a33011f8021","Type":"ContainerStarted","Data":"c4109df69ceb3934dac1533562ffd6424a69b30de3a7608c509412207acca750"} Nov 02 07:25:27.012619 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.012595 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" event={"ID":"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8","Type":"ContainerStarted","Data":"9445c0a7182bc4040ee2d5ade39bd08c2f5d4593ec47169ffdfd2666a9d725d3"} Nov 02 07:25:27.012713 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.012638 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" event={"ID":"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8","Type":"ContainerStarted","Data":"55bd77ae16c62ecec2c5ec20d2e027872af54164a14443a288faae1c434cdb09"} Nov 02 07:25:27.012713 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.012661 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" event={"ID":"f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8","Type":"ContainerStarted","Data":"b1f75cdd98adf62820c5235e632a89c1c04ef4b9c4cf0f11127ebaa277fc5691"} Nov 02 07:25:27.080956 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.080901 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.059921794 podStartE2EDuration="2.080888712s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.966918ms" isStatefulPod=false podCreationTimestamp="2026-11-02 07:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:25:27.079845948 +0000 UTC m=+27.378126772" watchObservedRunningTime="2026-11-02 07:25:27.080888712 +0000 UTC m=+27.379169536" Nov 02 07:25:27.128947 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.128897 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7946767c4b-g78ln" podStartSLOduration=2.082531919 podStartE2EDuration="4.128884679s" totalImagesPullingTime="2.04635276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:25:27.12860903 +0000 UTC m=+27.426889855" watchObservedRunningTime="2026-11-02 07:25:27.128884679 +0000 UTC m=+27.427165501" Nov 02 07:25:27.356861 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.356781 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5g4ck" Nov 02 07:25:27.472495 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.472459 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:25:27.606700 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.606669 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:25:27.765345 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.765301 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:25:27.899106 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:27.899073 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:25:30.078614 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:30.078576 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-25f6d" Nov 02 07:25:30.422382 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:30.422297 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:37.819699 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:37.819664 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cc9dd686b-z95lj"] Nov 02 07:25:43.138769 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:25:43.138728 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:25:44.099545 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:44.099506 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:44.115439 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:44.115413 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:45.095929 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:45.095889 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 07:25:54.714786 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:54.714738 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:25:54.855277 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:54.855244 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:25:59.636307 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:25:59.636271 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:02.198721 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:02.198683 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:02.332830 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:02.332793 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:02.506115 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:02.506086 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:02.637455 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:02.637414 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:02.839916 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:02.839789 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7cc9dd686b-z95lj" podUID="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" containerName="console" containerID="cri-o://3beb96bb221886bd05c82064c126d086524a4b1a10211337234edb6c061e1fc0" gracePeriod=15 Nov 02 07:26:03.073477 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.073449 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:26:03.124669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.124592 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-service-ca\") pod \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " Nov 02 07:26:03.124669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.124642 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-kube-api-access-dbk9t\" (UniqueName: \"kubernetes.io/projected/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-kube-api-access-dbk9t\") pod \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " Nov 02 07:26:03.124889 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.124701 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-trusted-ca-bundle\") pod \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " Nov 02 07:26:03.124889 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.124766 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-oauth-serving-cert\") pod \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " Nov 02 07:26:03.124889 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.124811 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-tmp\") pod \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " Nov 02 07:26:03.125049 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.124938 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-oauth-config\") pod \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " Nov 02 07:26:03.125049 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.125031 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-config\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-config\") pod \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " Nov 02 07:26:03.125149 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.125040 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-tmp" pod "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" (UID: "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:26:03.125149 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.125043 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-service-ca" pod "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" (UID: "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:26:03.125149 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.125077 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-serving-cert\") pod \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\" (UID: \"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02\") " Nov 02 07:26:03.125329 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.125199 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-trusted-ca-bundle" pod "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" (UID: "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:26:03.125419 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.125399 5899 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-tmp\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:26:03.125479 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.125426 5899 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:26:03.125479 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.125441 5899 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-trusted-ca-bundle\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:26:03.125479 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.125397 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-config" pod "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" (UID: "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:26:03.125676 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.125633 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-oauth-serving-cert" pod "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" (UID: "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:26:03.127457 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.127397 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-serving-cert" pod "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" (UID: "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:26:03.127560 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.127491 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-oauth-config" pod "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" (UID: "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:26:03.127606 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.127555 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-kube-api-access-dbk9t" pod "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" (UID: "b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02"). InnerVolumeSpecName "kube-api-access-dbk9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:26:03.156696 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.156658 5899 generic.go:356] "Generic (PLEG): container finished" podID="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" containerID="3beb96bb221886bd05c82064c126d086524a4b1a10211337234edb6c061e1fc0" exitCode=0 Nov 02 07:26:03.156815 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.156706 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc9dd686b-z95lj" event={"ID":"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02","Type":"ContainerDied","Data":"3beb96bb221886bd05c82064c126d086524a4b1a10211337234edb6c061e1fc0"} Nov 02 07:26:03.156815 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.156747 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc9dd686b-z95lj" event={"ID":"b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02","Type":"ContainerDied","Data":"792a2c51328c1027a6c630f5c72143d25ecd4946240c1adf01746da581a9a899"} Nov 02 07:26:03.156815 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.156767 5899 scope.go:122] "RemoveContainer" containerID="3beb96bb221886bd05c82064c126d086524a4b1a10211337234edb6c061e1fc0" Nov 02 07:26:03.156815 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.156706 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc9dd686b-z95lj" Nov 02 07:26:03.165374 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.165357 5899 scope.go:122] "RemoveContainer" containerID="3beb96bb221886bd05c82064c126d086524a4b1a10211337234edb6c061e1fc0" Nov 02 07:26:03.165639 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:26:03.165621 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3beb96bb221886bd05c82064c126d086524a4b1a10211337234edb6c061e1fc0\": container with ID starting with 3beb96bb221886bd05c82064c126d086524a4b1a10211337234edb6c061e1fc0 not found: ID does not exist" containerID="3beb96bb221886bd05c82064c126d086524a4b1a10211337234edb6c061e1fc0" Nov 02 07:26:03.165704 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.165653 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3beb96bb221886bd05c82064c126d086524a4b1a10211337234edb6c061e1fc0"} err="failed to get container status \"3beb96bb221886bd05c82064c126d086524a4b1a10211337234edb6c061e1fc0\": rpc error: code = NotFound desc = could not find container \"3beb96bb221886bd05c82064c126d086524a4b1a10211337234edb6c061e1fc0\": container with ID starting with 3beb96bb221886bd05c82064c126d086524a4b1a10211337234edb6c061e1fc0 not found: ID does not exist" Nov 02 07:26:03.215293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.215256 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cc9dd686b-z95lj"] Nov 02 07:26:03.221535 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.221504 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cc9dd686b-z95lj"] Nov 02 07:26:03.226827 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.226799 5899 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-serving-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:26:03.226827 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.226823 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbk9t\" (UniqueName: \"kubernetes.io/projected/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-kube-api-access-dbk9t\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:26:03.226999 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.226834 5899 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-oauth-serving-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:26:03.226999 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.226844 5899 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-oauth-config\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:26:03.226999 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.226854 5899 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02-console-config\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:26:03.816577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:03.816547 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" path="/var/lib/kubelet/pods/b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02/volumes" Nov 02 07:26:07.348359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:07.348324 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:07.478640 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:07.478608 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:13.139384 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:26:13.139349 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:26:14.026899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:14.026859 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:14.156944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:14.156898 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:43.139499 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:26:43.139415 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:26:51.159947 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:51.159909 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:53.485679 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:53.485641 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:53.621512 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:53.621484 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:59.760474 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:26:59.760439 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:27:13.139642 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:27:13.139602 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:27:17.810437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:27:17.810407 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:27:17.946336 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:27:17.946299 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:27:22.241931 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:27:22.241896 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:27:22.370622 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:27:22.370588 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:27:25.789474 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:27:25.789438 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:27:25.928306 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:27:25.928263 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:27:43.139140 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:27:43.139102 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:27:59.888924 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:27:59.888895 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:04.444808 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:04.444765 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:04.584759 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:04.584729 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:13.138624 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:28:13.138538 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:28:33.022766 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:33.022725 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:40.165709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:40.165680 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:40.350841 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:40.350808 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:40.497569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:40.497530 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:41.685108 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:41.685080 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:41.825516 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:41.825483 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:42.978209 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:42.978178 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:43.138778 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:28:43.138741 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:28:45.492199 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:45.492144 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:45.633454 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:45.633423 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:46.729673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:46.729607 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:46.878573 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:46.878541 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:47.021928 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.021833 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:47.243982 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.243941 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x"] Nov 02 07:28:47.244358 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.244341 5899 cpu_manager.go:414] "containerMap: removing container" podUID="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" containerName="console" Nov 02 07:28:47.244452 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.244359 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" containerName="console" Nov 02 07:28:47.244510 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.244477 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bcb2cc-73a5-4d9e-b0db-aaf5f3666c02" containerName="console" Nov 02 07:28:47.247602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.247582 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" Nov 02 07:28:47.276435 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.276355 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 07:28:47.276435 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.276361 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 07:28:47.276659 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.276466 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-n6wzt\"" Nov 02 07:28:47.277646 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.277621 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x"] Nov 02 07:28:47.314590 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.314560 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x\" (UID: \"e83d4792-a3d8-4dc5-8894-b8add99082fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" Nov 02 07:28:47.314748 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.314630 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv6hd\" (UniqueName: \"kubernetes.io/projected/e83d4792-a3d8-4dc5-8894-b8add99082fe-kube-api-access-wv6hd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x\" (UID: \"e83d4792-a3d8-4dc5-8894-b8add99082fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" Nov 02 07:28:47.314748 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.314729 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x\" (UID: \"e83d4792-a3d8-4dc5-8894-b8add99082fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" Nov 02 07:28:47.415683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.415641 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv6hd\" (UniqueName: \"kubernetes.io/projected/e83d4792-a3d8-4dc5-8894-b8add99082fe-kube-api-access-wv6hd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x\" (UID: \"e83d4792-a3d8-4dc5-8894-b8add99082fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" Nov 02 07:28:47.415867 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.415696 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x\" (UID: \"e83d4792-a3d8-4dc5-8894-b8add99082fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" Nov 02 07:28:47.415867 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.415760 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x\" (UID: \"e83d4792-a3d8-4dc5-8894-b8add99082fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" Nov 02 07:28:47.416190 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.416146 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x\" (UID: \"e83d4792-a3d8-4dc5-8894-b8add99082fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" Nov 02 07:28:47.416342 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.416321 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x\" (UID: \"e83d4792-a3d8-4dc5-8894-b8add99082fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" Nov 02 07:28:47.418631 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.418600 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk"] Nov 02 07:28:47.422136 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.422122 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" Nov 02 07:28:47.439847 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.439823 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-577xf\"" Nov 02 07:28:47.442652 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.442627 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv6hd\" (UniqueName: \"kubernetes.io/projected/e83d4792-a3d8-4dc5-8894-b8add99082fe-kube-api-access-wv6hd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x\" (UID: \"e83d4792-a3d8-4dc5-8894-b8add99082fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" Nov 02 07:28:47.443609 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.443587 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk"] Nov 02 07:28:47.516644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.516605 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk\" (UID: \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" Nov 02 07:28:47.516814 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.516734 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk\" (UID: \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" Nov 02 07:28:47.516814 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.516783 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbh9\" (UniqueName: \"kubernetes.io/projected/458da3e7-6123-4d1f-a8f2-4ad9814bc811-kube-api-access-fpbh9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk\" (UID: \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" Nov 02 07:28:47.563628 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.563541 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" Nov 02 07:28:47.617679 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.617648 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk\" (UID: \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" Nov 02 07:28:47.617823 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.617718 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk\" (UID: \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" Nov 02 07:28:47.617823 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.617751 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbh9\" (UniqueName: \"kubernetes.io/projected/458da3e7-6123-4d1f-a8f2-4ad9814bc811-kube-api-access-fpbh9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk\" (UID: \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" Nov 02 07:28:47.618092 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.618069 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk\" (UID: \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" Nov 02 07:28:47.618186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.618116 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk\" (UID: \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" Nov 02 07:28:47.642481 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.642453 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbh9\" (UniqueName: \"kubernetes.io/projected/458da3e7-6123-4d1f-a8f2-4ad9814bc811-kube-api-access-fpbh9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk\" (UID: \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" Nov 02 07:28:47.739573 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.739535 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" Nov 02 07:28:47.856406 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:47.856379 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x"] Nov 02 07:28:47.858285 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:28:47.858261 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83d4792_a3d8_4dc5_8894_b8add99082fe.slice/crio-8de3c792e578d32f5c4fe46588a48095792a99eedd1c0fde5efffa6a18a38525 WatchSource:0}: Error finding container 8de3c792e578d32f5c4fe46588a48095792a99eedd1c0fde5efffa6a18a38525: Status 404 returned error can't find the container with id 8de3c792e578d32f5c4fe46588a48095792a99eedd1c0fde5efffa6a18a38525 Nov 02 07:28:48.035306 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:48.035283 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk"] Nov 02 07:28:48.037266 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:28:48.037234 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458da3e7_6123_4d1f_a8f2_4ad9814bc811.slice/crio-9b8df364d737c8448273f35bebaeff559008ab8568dafd51a233baaf12b6226a WatchSource:0}: Error finding container 9b8df364d737c8448273f35bebaeff559008ab8568dafd51a233baaf12b6226a: Status 404 returned error can't find the container with id 9b8df364d737c8448273f35bebaeff559008ab8568dafd51a233baaf12b6226a Nov 02 07:28:48.763883 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:48.763841 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" event={"ID":"458da3e7-6123-4d1f-a8f2-4ad9814bc811","Type":"ContainerStarted","Data":"9b8df364d737c8448273f35bebaeff559008ab8568dafd51a233baaf12b6226a"} Nov 02 07:28:48.766216 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:48.765469 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" event={"ID":"e83d4792-a3d8-4dc5-8894-b8add99082fe","Type":"ContainerStarted","Data":"8de3c792e578d32f5c4fe46588a48095792a99eedd1c0fde5efffa6a18a38525"} Nov 02 07:28:50.042581 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:50.042547 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:28:54.792301 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:54.792264 5899 generic.go:356] "Generic (PLEG): container finished" podID="e83d4792-a3d8-4dc5-8894-b8add99082fe" containerID="c58dcdd899c78c1b1aae0ef2daf5214079dd87e7117136ad4f520eee58ed3524" exitCode=0 Nov 02 07:28:54.792787 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:54.792356 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" event={"ID":"e83d4792-a3d8-4dc5-8894-b8add99082fe","Type":"ContainerDied","Data":"c58dcdd899c78c1b1aae0ef2daf5214079dd87e7117136ad4f520eee58ed3524"} Nov 02 07:28:54.793896 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:54.793875 5899 generic.go:356] "Generic (PLEG): container finished" podID="458da3e7-6123-4d1f-a8f2-4ad9814bc811" containerID="32c249c8f4cd4c2a46256d5ebac04d30ffb2c5f6c1b52e1f7bfa532f8fb68ea2" exitCode=0 Nov 02 07:28:54.793997 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:54.793932 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" event={"ID":"458da3e7-6123-4d1f-a8f2-4ad9814bc811","Type":"ContainerDied","Data":"32c249c8f4cd4c2a46256d5ebac04d30ffb2c5f6c1b52e1f7bfa532f8fb68ea2"} Nov 02 07:28:57.808115 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:57.808080 5899 generic.go:356] "Generic (PLEG): container finished" podID="e83d4792-a3d8-4dc5-8894-b8add99082fe" containerID="a8053adccc448d51bca3b5f1aad96a5f87d7af2fc87d0fd53e13364b7aa4a153" exitCode=0 Nov 02 07:28:57.808586 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:57.808182 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" event={"ID":"e83d4792-a3d8-4dc5-8894-b8add99082fe","Type":"ContainerDied","Data":"a8053adccc448d51bca3b5f1aad96a5f87d7af2fc87d0fd53e13364b7aa4a153"} Nov 02 07:28:57.810032 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:57.810017 5899 generic.go:356] "Generic (PLEG): container finished" podID="458da3e7-6123-4d1f-a8f2-4ad9814bc811" containerID="e2d6e7f2b25d65735200d0a0a0399a09e3d4475a3b3454012b850d1972f08f9c" exitCode=0 Nov 02 07:28:57.810105 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:28:57.810063 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" event={"ID":"458da3e7-6123-4d1f-a8f2-4ad9814bc811","Type":"ContainerDied","Data":"e2d6e7f2b25d65735200d0a0a0399a09e3d4475a3b3454012b850d1972f08f9c"} Nov 02 07:29:00.021625 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:00.021588 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:04.843085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:04.843007 5899 generic.go:356] "Generic (PLEG): container finished" podID="458da3e7-6123-4d1f-a8f2-4ad9814bc811" containerID="8f6bb11ade2f01de33bf1a04cf9d51abc442fa2a2e468e85a28a33f31e584968" exitCode=0 Nov 02 07:29:04.843533 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:04.843095 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" event={"ID":"458da3e7-6123-4d1f-a8f2-4ad9814bc811","Type":"ContainerDied","Data":"8f6bb11ade2f01de33bf1a04cf9d51abc442fa2a2e468e85a28a33f31e584968"} Nov 02 07:29:04.845100 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:04.845077 5899 generic.go:356] "Generic (PLEG): container finished" podID="e83d4792-a3d8-4dc5-8894-b8add99082fe" containerID="675349fe529441dbcea4b3e18e8bc9f7bb65c5d81821ea5b9d313a96369640c3" exitCode=0 Nov 02 07:29:04.845214 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:04.845102 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" event={"ID":"e83d4792-a3d8-4dc5-8894-b8add99082fe","Type":"ContainerDied","Data":"675349fe529441dbcea4b3e18e8bc9f7bb65c5d81821ea5b9d313a96369640c3"} Nov 02 07:29:05.973363 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:05.973337 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" Nov 02 07:29:06.005259 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.005238 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" Nov 02 07:29:06.096290 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.096259 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-util\" (UniqueName: \"kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-util\") pod \"e83d4792-a3d8-4dc5-8894-b8add99082fe\" (UID: \"e83d4792-a3d8-4dc5-8894-b8add99082fe\") " Nov 02 07:29:06.096473 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.096299 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/458da3e7-6123-4d1f-a8f2-4ad9814bc811-kube-api-access-fpbh9\" (UniqueName: \"kubernetes.io/projected/458da3e7-6123-4d1f-a8f2-4ad9814bc811-kube-api-access-fpbh9\") pod \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\" (UID: \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\") " Nov 02 07:29:06.096473 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.096329 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-bundle\" (UniqueName: \"kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-bundle\") pod \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\" (UID: \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\") " Nov 02 07:29:06.096473 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.096390 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-bundle\") pod \"e83d4792-a3d8-4dc5-8894-b8add99082fe\" (UID: \"e83d4792-a3d8-4dc5-8894-b8add99082fe\") " Nov 02 07:29:06.096473 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.096409 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e83d4792-a3d8-4dc5-8894-b8add99082fe-kube-api-access-wv6hd\" (UniqueName: \"kubernetes.io/projected/e83d4792-a3d8-4dc5-8894-b8add99082fe-kube-api-access-wv6hd\") pod \"e83d4792-a3d8-4dc5-8894-b8add99082fe\" (UID: \"e83d4792-a3d8-4dc5-8894-b8add99082fe\") " Nov 02 07:29:06.096473 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.096444 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-util\" (UniqueName: \"kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-util\") pod \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\" (UID: \"458da3e7-6123-4d1f-a8f2-4ad9814bc811\") " Nov 02 07:29:06.096884 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.096759 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-bundle" pod "e83d4792-a3d8-4dc5-8894-b8add99082fe" (UID: "e83d4792-a3d8-4dc5-8894-b8add99082fe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:29:06.097066 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.097038 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-bundle" pod "458da3e7-6123-4d1f-a8f2-4ad9814bc811" (UID: "458da3e7-6123-4d1f-a8f2-4ad9814bc811"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:29:06.098674 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.098640 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83d4792-a3d8-4dc5-8894-b8add99082fe-kube-api-access-wv6hd" pod "e83d4792-a3d8-4dc5-8894-b8add99082fe" (UID: "e83d4792-a3d8-4dc5-8894-b8add99082fe"). InnerVolumeSpecName "kube-api-access-wv6hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:29:06.098674 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.098656 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458da3e7-6123-4d1f-a8f2-4ad9814bc811-kube-api-access-fpbh9" pod "458da3e7-6123-4d1f-a8f2-4ad9814bc811" (UID: "458da3e7-6123-4d1f-a8f2-4ad9814bc811"). InnerVolumeSpecName "kube-api-access-fpbh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:29:06.100535 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.100483 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-util" pod "e83d4792-a3d8-4dc5-8894-b8add99082fe" (UID: "e83d4792-a3d8-4dc5-8894-b8add99082fe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:29:06.102189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.102149 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-util" pod "458da3e7-6123-4d1f-a8f2-4ad9814bc811" (UID: "458da3e7-6123-4d1f-a8f2-4ad9814bc811"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:29:06.197639 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.197601 5899 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-util\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:29:06.197639 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.197632 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpbh9\" (UniqueName: \"kubernetes.io/projected/458da3e7-6123-4d1f-a8f2-4ad9814bc811-kube-api-access-fpbh9\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:29:06.197639 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.197643 5899 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-bundle\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:29:06.197639 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.197652 5899 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e83d4792-a3d8-4dc5-8894-b8add99082fe-bundle\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:29:06.197896 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.197660 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv6hd\" (UniqueName: \"kubernetes.io/projected/e83d4792-a3d8-4dc5-8894-b8add99082fe-kube-api-access-wv6hd\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:29:06.197896 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.197668 5899 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/458da3e7-6123-4d1f-a8f2-4ad9814bc811-util\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:29:06.855302 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.855275 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" Nov 02 07:29:06.855302 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.855292 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448zr5x" event={"ID":"e83d4792-a3d8-4dc5-8894-b8add99082fe","Type":"ContainerDied","Data":"8de3c792e578d32f5c4fe46588a48095792a99eedd1c0fde5efffa6a18a38525"} Nov 02 07:29:06.855547 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.855341 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8de3c792e578d32f5c4fe46588a48095792a99eedd1c0fde5efffa6a18a38525" Nov 02 07:29:06.857020 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.856998 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" event={"ID":"458da3e7-6123-4d1f-a8f2-4ad9814bc811","Type":"ContainerDied","Data":"9b8df364d737c8448273f35bebaeff559008ab8568dafd51a233baaf12b6226a"} Nov 02 07:29:06.857020 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.857020 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cnj2xk" Nov 02 07:29:06.857173 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:06.857031 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b8df364d737c8448273f35bebaeff559008ab8568dafd51a233baaf12b6226a" Nov 02 07:29:07.526011 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:07.525970 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:07.663699 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:07.663665 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:09.172758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.172724 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd"] Nov 02 07:29:09.173240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173111 5899 cpu_manager.go:414] "containerMap: removing container" podUID="e83d4792-a3d8-4dc5-8894-b8add99082fe" containerName="util" Nov 02 07:29:09.173240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173128 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e83d4792-a3d8-4dc5-8894-b8add99082fe" containerName="util" Nov 02 07:29:09.173240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173137 5899 cpu_manager.go:414] "containerMap: removing container" podUID="458da3e7-6123-4d1f-a8f2-4ad9814bc811" containerName="pull" Nov 02 07:29:09.173240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173143 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="458da3e7-6123-4d1f-a8f2-4ad9814bc811" containerName="pull" Nov 02 07:29:09.173240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173149 5899 cpu_manager.go:414] "containerMap: removing container" podUID="e83d4792-a3d8-4dc5-8894-b8add99082fe" containerName="extract" Nov 02 07:29:09.173240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173153 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e83d4792-a3d8-4dc5-8894-b8add99082fe" containerName="extract" Nov 02 07:29:09.173240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173176 5899 cpu_manager.go:414] "containerMap: removing container" podUID="458da3e7-6123-4d1f-a8f2-4ad9814bc811" containerName="extract" Nov 02 07:29:09.173240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173181 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="458da3e7-6123-4d1f-a8f2-4ad9814bc811" containerName="extract" Nov 02 07:29:09.173240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173190 5899 cpu_manager.go:414] "containerMap: removing container" podUID="e83d4792-a3d8-4dc5-8894-b8add99082fe" containerName="pull" Nov 02 07:29:09.173240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173196 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e83d4792-a3d8-4dc5-8894-b8add99082fe" containerName="pull" Nov 02 07:29:09.173240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173225 5899 cpu_manager.go:414] "containerMap: removing container" podUID="458da3e7-6123-4d1f-a8f2-4ad9814bc811" containerName="util" Nov 02 07:29:09.173240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173231 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="458da3e7-6123-4d1f-a8f2-4ad9814bc811" containerName="util" Nov 02 07:29:09.173591 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173300 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="e83d4792-a3d8-4dc5-8894-b8add99082fe" containerName="extract" Nov 02 07:29:09.173591 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.173308 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="458da3e7-6123-4d1f-a8f2-4ad9814bc811" containerName="extract" Nov 02 07:29:09.177848 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.177829 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd" Nov 02 07:29:09.209059 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.209030 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 07:29:09.209059 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.209039 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 07:29:09.209279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.209071 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-4drj7\"" Nov 02 07:29:09.209921 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.209902 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd"] Nov 02 07:29:09.323466 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.323433 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24fb4291-c9a6-45c6-adfe-c851b3fd55c0-tmp\") pod \"jobset-operator-674cd59dc5-59bgd\" (UID: \"24fb4291-c9a6-45c6-adfe-c851b3fd55c0\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd" Nov 02 07:29:09.323623 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.323483 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm5rf\" (UniqueName: \"kubernetes.io/projected/24fb4291-c9a6-45c6-adfe-c851b3fd55c0-kube-api-access-rm5rf\") pod \"jobset-operator-674cd59dc5-59bgd\" (UID: \"24fb4291-c9a6-45c6-adfe-c851b3fd55c0\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd" Nov 02 07:29:09.424340 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.424266 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm5rf\" (UniqueName: \"kubernetes.io/projected/24fb4291-c9a6-45c6-adfe-c851b3fd55c0-kube-api-access-rm5rf\") pod \"jobset-operator-674cd59dc5-59bgd\" (UID: \"24fb4291-c9a6-45c6-adfe-c851b3fd55c0\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd" Nov 02 07:29:09.424504 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.424342 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24fb4291-c9a6-45c6-adfe-c851b3fd55c0-tmp\") pod \"jobset-operator-674cd59dc5-59bgd\" (UID: \"24fb4291-c9a6-45c6-adfe-c851b3fd55c0\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd" Nov 02 07:29:09.424701 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.424685 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24fb4291-c9a6-45c6-adfe-c851b3fd55c0-tmp\") pod \"jobset-operator-674cd59dc5-59bgd\" (UID: \"24fb4291-c9a6-45c6-adfe-c851b3fd55c0\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd" Nov 02 07:29:09.448774 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.448745 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm5rf\" (UniqueName: \"kubernetes.io/projected/24fb4291-c9a6-45c6-adfe-c851b3fd55c0-kube-api-access-rm5rf\") pod \"jobset-operator-674cd59dc5-59bgd\" (UID: \"24fb4291-c9a6-45c6-adfe-c851b3fd55c0\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd" Nov 02 07:29:09.507659 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.507625 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd" Nov 02 07:29:09.822747 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.822723 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd"] Nov 02 07:29:09.824644 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:29:09.824619 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24fb4291_c9a6_45c6_adfe_c851b3fd55c0.slice/crio-404735cc03503fab42e9d8f71e09c8b3b0eeba154b36f93425c886adf7c85b6d WatchSource:0}: Error finding container 404735cc03503fab42e9d8f71e09c8b3b0eeba154b36f93425c886adf7c85b6d: Status 404 returned error can't find the container with id 404735cc03503fab42e9d8f71e09c8b3b0eeba154b36f93425c886adf7c85b6d Nov 02 07:29:09.872120 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:09.872088 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd" event={"ID":"24fb4291-c9a6-45c6-adfe-c851b3fd55c0","Type":"ContainerStarted","Data":"404735cc03503fab42e9d8f71e09c8b3b0eeba154b36f93425c886adf7c85b6d"} Nov 02 07:29:12.887287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:12.887250 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd" event={"ID":"24fb4291-c9a6-45c6-adfe-c851b3fd55c0","Type":"ContainerStarted","Data":"6a63ba944e1add66cc9eae90792d0139919b24895024ab3f2be835f39a428b67"} Nov 02 07:29:12.930522 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:12.930446 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59bgd" podStartSLOduration=1.4285535409999999 podStartE2EDuration="3.930431597s" totalImagesPullingTime="2.501878056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:29:12.929788083 +0000 UTC m=+253.228068908" watchObservedRunningTime="2026-11-02 07:29:12.930431597 +0000 UTC m=+253.228712425" Nov 02 07:29:13.141735 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:29:13.141641 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:29:13.397846 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:13.397759 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:17.339643 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:17.339602 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:19.015207 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:19.015150 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:19.153648 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:19.153596 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:21.534771 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:21.534740 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:21.690741 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:21.690714 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:21.838258 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:21.838180 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v"] Nov 02 07:29:21.840382 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:21.840361 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v" Nov 02 07:29:21.872838 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:21.872815 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 07:29:21.872967 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:21.872890 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-tkz5v\"" Nov 02 07:29:21.872967 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:21.872891 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 07:29:21.873885 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:21.873867 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v"] Nov 02 07:29:21.921911 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:21.921881 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpl9w\" (UniqueName: \"kubernetes.io/projected/f7dcb101-c56d-4182-abc0-ce4e6f9ce0df-kube-api-access-qpl9w\") pod \"cert-manager-webhook-5dbbbf75cc-q7s4v\" (UID: \"f7dcb101-c56d-4182-abc0-ce4e6f9ce0df\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v" Nov 02 07:29:21.922050 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:21.921917 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7dcb101-c56d-4182-abc0-ce4e6f9ce0df-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-q7s4v\" (UID: \"f7dcb101-c56d-4182-abc0-ce4e6f9ce0df\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v" Nov 02 07:29:22.022260 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:22.022225 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpl9w\" (UniqueName: \"kubernetes.io/projected/f7dcb101-c56d-4182-abc0-ce4e6f9ce0df-kube-api-access-qpl9w\") pod \"cert-manager-webhook-5dbbbf75cc-q7s4v\" (UID: \"f7dcb101-c56d-4182-abc0-ce4e6f9ce0df\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v" Nov 02 07:29:22.022260 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:22.022264 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7dcb101-c56d-4182-abc0-ce4e6f9ce0df-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-q7s4v\" (UID: \"f7dcb101-c56d-4182-abc0-ce4e6f9ce0df\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v" Nov 02 07:29:22.046466 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:22.046432 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7dcb101-c56d-4182-abc0-ce4e6f9ce0df-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-q7s4v\" (UID: \"f7dcb101-c56d-4182-abc0-ce4e6f9ce0df\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v" Nov 02 07:29:22.046596 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:22.046507 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpl9w\" (UniqueName: \"kubernetes.io/projected/f7dcb101-c56d-4182-abc0-ce4e6f9ce0df-kube-api-access-qpl9w\") pod \"cert-manager-webhook-5dbbbf75cc-q7s4v\" (UID: \"f7dcb101-c56d-4182-abc0-ce4e6f9ce0df\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v" Nov 02 07:29:22.149875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:22.149795 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v" Nov 02 07:29:22.191533 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:22.191502 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:22.449180 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:22.449140 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v"] Nov 02 07:29:22.450754 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:29:22.450728 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7dcb101_c56d_4182_abc0_ce4e6f9ce0df.slice/crio-7d55ff6062fa91d34932bf5363fab7a8c605f09d54560426f07d4fc9573d5c15 WatchSource:0}: Error finding container 7d55ff6062fa91d34932bf5363fab7a8c605f09d54560426f07d4fc9573d5c15: Status 404 returned error can't find the container with id 7d55ff6062fa91d34932bf5363fab7a8c605f09d54560426f07d4fc9573d5c15 Nov 02 07:29:22.940556 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:22.940501 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v" event={"ID":"f7dcb101-c56d-4182-abc0-ce4e6f9ce0df","Type":"ContainerStarted","Data":"7d55ff6062fa91d34932bf5363fab7a8c605f09d54560426f07d4fc9573d5c15"} Nov 02 07:29:23.977116 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:23.977086 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:24.157812 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:24.157779 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:24.950715 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:24.950677 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v" event={"ID":"f7dcb101-c56d-4182-abc0-ce4e6f9ce0df","Type":"ContainerStarted","Data":"72e82c7ba677d41cc6f62bdd49fab5d64305d6a1254e93ee7fb021b2fb21a48a"} Nov 02 07:29:24.950862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:24.950741 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v" Nov 02 07:29:24.993676 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:24.993613 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v" podStartSLOduration=1.589299193 podStartE2EDuration="3.99359789s" totalImagesPullingTime="2.404298697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:29:24.993451573 +0000 UTC m=+265.291732411" watchObservedRunningTime="2026-11-02 07:29:24.99359789 +0000 UTC m=+265.291878717" Nov 02 07:29:26.199528 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:26.199480 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:26.996461 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:26.996425 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:28.221621 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.221591 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66"] Nov 02 07:29:28.223686 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.223670 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66" Nov 02 07:29:28.252230 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.252198 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:28.261853 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.261832 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dhttw\"" Nov 02 07:29:28.263086 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.263065 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66"] Nov 02 07:29:28.283476 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.283449 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5766eae2-110b-45a2-86e6-edf064ff87a4-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-ztg66\" (UID: \"5766eae2-110b-45a2-86e6-edf064ff87a4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66" Nov 02 07:29:28.283584 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.283500 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb8pz\" (UniqueName: \"kubernetes.io/projected/5766eae2-110b-45a2-86e6-edf064ff87a4-kube-api-access-lb8pz\") pod \"cert-manager-cainjector-5c57bbd4fc-ztg66\" (UID: \"5766eae2-110b-45a2-86e6-edf064ff87a4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66" Nov 02 07:29:28.384965 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.384931 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5766eae2-110b-45a2-86e6-edf064ff87a4-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-ztg66\" (UID: \"5766eae2-110b-45a2-86e6-edf064ff87a4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66" Nov 02 07:29:28.385116 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.384979 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb8pz\" (UniqueName: \"kubernetes.io/projected/5766eae2-110b-45a2-86e6-edf064ff87a4-kube-api-access-lb8pz\") pod \"cert-manager-cainjector-5c57bbd4fc-ztg66\" (UID: \"5766eae2-110b-45a2-86e6-edf064ff87a4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66" Nov 02 07:29:28.393493 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.393458 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:28.415441 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.415410 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5766eae2-110b-45a2-86e6-edf064ff87a4-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-ztg66\" (UID: \"5766eae2-110b-45a2-86e6-edf064ff87a4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66" Nov 02 07:29:28.415532 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.415482 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb8pz\" (UniqueName: \"kubernetes.io/projected/5766eae2-110b-45a2-86e6-edf064ff87a4-kube-api-access-lb8pz\") pod \"cert-manager-cainjector-5c57bbd4fc-ztg66\" (UID: \"5766eae2-110b-45a2-86e6-edf064ff87a4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66" Nov 02 07:29:28.533735 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.533704 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66" Nov 02 07:29:28.598510 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.598475 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:28.744279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.744241 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:28.825386 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.825361 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66"] Nov 02 07:29:28.826528 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:29:28.826502 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5766eae2_110b_45a2_86e6_edf064ff87a4.slice/crio-0e3610fab820f065be36349d7bb985e910a30dbf53751bcafb3781f1231e29d0 WatchSource:0}: Error finding container 0e3610fab820f065be36349d7bb985e910a30dbf53751bcafb3781f1231e29d0: Status 404 returned error can't find the container with id 0e3610fab820f065be36349d7bb985e910a30dbf53751bcafb3781f1231e29d0 Nov 02 07:29:28.969042 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.969004 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66" event={"ID":"5766eae2-110b-45a2-86e6-edf064ff87a4","Type":"ContainerStarted","Data":"d07ba208b4e1b95e2258a3fbf74fed5665cd653446724f4e806d3c4ce84a5296"} Nov 02 07:29:28.969226 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:28.969052 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66" event={"ID":"5766eae2-110b-45a2-86e6-edf064ff87a4","Type":"ContainerStarted","Data":"0e3610fab820f065be36349d7bb985e910a30dbf53751bcafb3781f1231e29d0"} Nov 02 07:29:29.014496 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:29.014443 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ztg66" podStartSLOduration=1.014428604 podStartE2EDuration="1.014428604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:29:29.014122628 +0000 UTC m=+269.312403452" watchObservedRunningTime="2026-11-02 07:29:29.014428604 +0000 UTC m=+269.312709429" Nov 02 07:29:29.893780 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:29.893749 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:30.033554 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:30.033519 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:30.957871 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:30.957840 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-q7s4v" Nov 02 07:29:31.609344 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:31.609308 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:31.850120 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:31.850057 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:32.061752 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:32.061723 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:33.110984 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.110950 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-bjbfh"] Nov 02 07:29:33.113152 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.113135 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-bjbfh" Nov 02 07:29:33.142896 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.142861 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-chmm6\"" Nov 02 07:29:33.143831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.143814 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-bjbfh"] Nov 02 07:29:33.224992 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.224950 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltp5q\" (UniqueName: \"kubernetes.io/projected/3dc8da40-6235-454b-94c6-dec3dfc5c979-kube-api-access-ltp5q\") pod \"cert-manager-597b95cb84-bjbfh\" (UID: \"3dc8da40-6235-454b-94c6-dec3dfc5c979\") " pod="cert-manager/cert-manager-597b95cb84-bjbfh" Nov 02 07:29:33.224992 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.224992 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3dc8da40-6235-454b-94c6-dec3dfc5c979-bound-sa-token\") pod \"cert-manager-597b95cb84-bjbfh\" (UID: \"3dc8da40-6235-454b-94c6-dec3dfc5c979\") " pod="cert-manager/cert-manager-597b95cb84-bjbfh" Nov 02 07:29:33.326173 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.326124 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltp5q\" (UniqueName: \"kubernetes.io/projected/3dc8da40-6235-454b-94c6-dec3dfc5c979-kube-api-access-ltp5q\") pod \"cert-manager-597b95cb84-bjbfh\" (UID: \"3dc8da40-6235-454b-94c6-dec3dfc5c979\") " pod="cert-manager/cert-manager-597b95cb84-bjbfh" Nov 02 07:29:33.326350 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.326197 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3dc8da40-6235-454b-94c6-dec3dfc5c979-bound-sa-token\") pod \"cert-manager-597b95cb84-bjbfh\" (UID: \"3dc8da40-6235-454b-94c6-dec3dfc5c979\") " pod="cert-manager/cert-manager-597b95cb84-bjbfh" Nov 02 07:29:33.350583 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.350555 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3dc8da40-6235-454b-94c6-dec3dfc5c979-bound-sa-token\") pod \"cert-manager-597b95cb84-bjbfh\" (UID: \"3dc8da40-6235-454b-94c6-dec3dfc5c979\") " pod="cert-manager/cert-manager-597b95cb84-bjbfh" Nov 02 07:29:33.350796 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.350776 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltp5q\" (UniqueName: \"kubernetes.io/projected/3dc8da40-6235-454b-94c6-dec3dfc5c979-kube-api-access-ltp5q\") pod \"cert-manager-597b95cb84-bjbfh\" (UID: \"3dc8da40-6235-454b-94c6-dec3dfc5c979\") " pod="cert-manager/cert-manager-597b95cb84-bjbfh" Nov 02 07:29:33.423357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.423272 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-bjbfh" Nov 02 07:29:33.714299 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.714270 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-bjbfh"] Nov 02 07:29:33.715489 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:29:33.715460 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dc8da40_6235_454b_94c6_dec3dfc5c979.slice/crio-771143c61a2cf6241aa1076b004b565353c0828d13a7ae0875d13d1be48f755f WatchSource:0}: Error finding container 771143c61a2cf6241aa1076b004b565353c0828d13a7ae0875d13d1be48f755f: Status 404 returned error can't find the container with id 771143c61a2cf6241aa1076b004b565353c0828d13a7ae0875d13d1be48f755f Nov 02 07:29:33.990541 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.990505 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-bjbfh" event={"ID":"3dc8da40-6235-454b-94c6-dec3dfc5c979","Type":"ContainerStarted","Data":"1db06be076f07d83b9d043088416b8e30588ed55de7e0b7fe00bc29a6ebd76f1"} Nov 02 07:29:33.990541 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:33.990546 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-bjbfh" event={"ID":"3dc8da40-6235-454b-94c6-dec3dfc5c979","Type":"ContainerStarted","Data":"771143c61a2cf6241aa1076b004b565353c0828d13a7ae0875d13d1be48f755f"} Nov 02 07:29:34.033589 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:34.033536 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-bjbfh" podStartSLOduration=1.033523848 podStartE2EDuration="1.033523848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:29:34.03312682 +0000 UTC m=+274.331407644" watchObservedRunningTime="2026-11-02 07:29:34.033523848 +0000 UTC m=+274.331804673" Nov 02 07:29:35.004533 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:35.004497 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:35.597832 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:35.597803 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:36.455719 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:36.455674 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:36.655582 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:36.655527 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:37.058830 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:37.058800 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:37.248888 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:37.248822 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:38.563974 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.563944 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:38.799226 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.799193 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm"] Nov 02 07:29:38.803952 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.803931 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.821863 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.821805 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm"] Nov 02 07:29:38.873003 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.872944 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-webhook-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.873003 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.872990 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-tmp\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.873283 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.873208 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-metrics-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.873283 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.873236 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-manifests\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.873283 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.873257 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.873415 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.873331 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llpj6\" (UniqueName: \"kubernetes.io/projected/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-kube-api-access-llpj6\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.884448 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.884420 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 07:29:38.884775 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.884758 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 07:29:38.885013 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.884999 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jkbxf\"" Nov 02 07:29:38.885240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.885222 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 07:29:38.885540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.885522 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 07:29:38.885634 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.885621 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 07:29:38.967385 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.967353 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:38.974492 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.974469 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-metrics-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.974562 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.974496 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-manifests\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.974562 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.974515 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.974656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.974637 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llpj6\" (UniqueName: \"kubernetes.io/projected/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-kube-api-access-llpj6\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.974720 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.974703 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-webhook-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.974759 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.974740 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-tmp\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.974948 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.974931 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-manifests\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.976980 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.976952 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-tmp\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.977098 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.977081 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.977147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.977095 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-webhook-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:38.977423 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:38.977406 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-metrics-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:39.002563 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:39.002541 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llpj6\" (UniqueName: \"kubernetes.io/projected/eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1-kube-api-access-llpj6\") pod \"opendatahub-operator-controller-manager-756f6c987b-764mm\" (UID: \"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:39.110441 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:39.110359 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:39.121298 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:39.121270 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:39.296352 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:39.296318 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:39.439021 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:39.438935 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:39.469728 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:39.469701 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm"] Nov 02 07:29:39.470299 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:29:39.470273 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecab1a5_2cd8_4ee3_a1ba_dbde53dfa0b1.slice/crio-fb39f31828429da93fb79c895156c49bff7ae1d80368221f8087fd8b7bbca900 WatchSource:0}: Error finding container fb39f31828429da93fb79c895156c49bff7ae1d80368221f8087fd8b7bbca900: Status 404 returned error can't find the container with id fb39f31828429da93fb79c895156c49bff7ae1d80368221f8087fd8b7bbca900 Nov 02 07:29:40.016394 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:40.016360 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" event={"ID":"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1","Type":"ContainerStarted","Data":"fb39f31828429da93fb79c895156c49bff7ae1d80368221f8087fd8b7bbca900"} Nov 02 07:29:43.034595 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:43.034556 5899 generic.go:356] "Generic (PLEG): container finished" podID="eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1" containerID="b41913a99c9ee7ddc0671db7dec4e8c27495b370b643e8131a962da291bcb874" exitCode=0 Nov 02 07:29:43.034963 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:43.034665 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" event={"ID":"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1","Type":"ContainerDied","Data":"b41913a99c9ee7ddc0671db7dec4e8c27495b370b643e8131a962da291bcb874"} Nov 02 07:29:43.139337 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:29:43.139304 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:29:43.307837 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:43.307805 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:43.463003 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:43.462971 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:44.027795 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:44.027759 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:44.040936 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:44.040910 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" event={"ID":"eecab1a5-2cd8-4ee3-a1ba-dbde53dfa0b1","Type":"ContainerStarted","Data":"d2738a1995153cbd90b14d9431bb5d7f2270d69f8a3166b263e8fd5929d294f9"} Nov 02 07:29:44.041360 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:44.041050 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:44.137095 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:44.137047 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" podStartSLOduration=2.444411038 podStartE2EDuration="6.137033135s" totalImagesPullingTime="3.196776774s" totalInitContainerRuntime="495.845323ms" isStatefulPod=false podCreationTimestamp="2026-11-02 07:29:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:29:44.123257761 +0000 UTC m=+284.421538583" watchObservedRunningTime="2026-11-02 07:29:44.137033135 +0000 UTC m=+284.435313961" Nov 02 07:29:44.176985 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:44.176954 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:48.397452 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.397414 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52"] Nov 02 07:29:48.403853 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.403831 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" Nov 02 07:29:48.421150 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.421124 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52"] Nov 02 07:29:48.429097 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.429073 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:48.433576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.433558 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 07:29:48.436203 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.436187 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 07:29:48.436282 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.436266 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 07:29:48.436328 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.436294 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tbgzz\"" Nov 02 07:29:48.436368 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.436329 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 07:29:48.568854 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.568800 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62j28\" (UniqueName: \"kubernetes.io/projected/b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae-kube-api-access-62j28\") pod \"kuadrant-console-plugin-66c4fb575d-xjq52\" (UID: \"b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" Nov 02 07:29:48.569020 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.568926 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-xjq52\" (UID: \"b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" Nov 02 07:29:48.569020 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.568981 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-xjq52\" (UID: \"b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" Nov 02 07:29:48.575330 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.575304 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:48.669779 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.669702 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-xjq52\" (UID: \"b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" Nov 02 07:29:48.669899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.669778 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-xjq52\" (UID: \"b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" Nov 02 07:29:48.669899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.669819 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62j28\" (UniqueName: \"kubernetes.io/projected/b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae-kube-api-access-62j28\") pod \"kuadrant-console-plugin-66c4fb575d-xjq52\" (UID: \"b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" Nov 02 07:29:48.670496 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.670479 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-xjq52\" (UID: \"b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" Nov 02 07:29:48.672304 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.672283 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-xjq52\" (UID: \"b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" Nov 02 07:29:48.694452 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.694424 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62j28\" (UniqueName: \"kubernetes.io/projected/b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae-kube-api-access-62j28\") pod \"kuadrant-console-plugin-66c4fb575d-xjq52\" (UID: \"b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" Nov 02 07:29:48.715227 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:48.715200 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" Nov 02 07:29:49.013795 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:49.013767 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52"] Nov 02 07:29:49.014522 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:29:49.014500 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0407d3f_b9ba_4445_97ce_cb4ceb2e77ae.slice/crio-075b0b5f6fdd9c338621b868739b18600debd4d24e2ec32558cb44951abd34cd WatchSource:0}: Error finding container 075b0b5f6fdd9c338621b868739b18600debd4d24e2ec32558cb44951abd34cd: Status 404 returned error can't find the container with id 075b0b5f6fdd9c338621b868739b18600debd4d24e2ec32558cb44951abd34cd Nov 02 07:29:49.062758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:49.062720 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" event={"ID":"b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae","Type":"ContainerStarted","Data":"075b0b5f6fdd9c338621b868739b18600debd4d24e2ec32558cb44951abd34cd"} Nov 02 07:29:49.402536 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:49.402435 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:49.539079 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:49.539042 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:49.790783 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:49.790651 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:49.936866 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:49.936773 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:53.387704 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:53.387671 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:53.523564 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:53.523526 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:54.089000 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:54.088962 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" event={"ID":"b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae","Type":"ContainerStarted","Data":"bf6f52618a1446aa645b625b8356a7b42eeaa3b814ad6fc0f61eb4cb69ed926b"} Nov 02 07:29:54.132469 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:54.132419 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xjq52" podStartSLOduration=1.583157731 podStartE2EDuration="6.132406093s" totalImagesPullingTime="4.549248362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:29:54.131559686 +0000 UTC m=+294.429840514" watchObservedRunningTime="2026-11-02 07:29:54.132406093 +0000 UTC m=+294.430686952" Nov 02 07:29:54.951780 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:54.951743 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:55.047422 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:55.047388 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-764mm" Nov 02 07:29:55.095574 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:55.095543 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:57.845327 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:57.845293 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:57.986069 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:57.986013 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:59.622180 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:59.622130 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:29:59.765105 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:29:59.765075 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:00.140199 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:00.140155 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:06.247569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:06.247534 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:06.400251 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:06.400208 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:06.548352 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:06.548268 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:06.694345 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:06.694312 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:06.981220 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:06.981186 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:07.129739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.129706 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg"] Nov 02 07:30:07.134262 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.134238 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" Nov 02 07:30:07.157938 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.157905 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:07.167320 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.167296 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 07:30:07.167516 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.167336 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 07:30:07.168636 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.168609 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg"] Nov 02 07:30:07.219177 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.219128 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 07:30:07.219371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.219200 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 07:30:07.219371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.219327 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jmlf4\"" Nov 02 07:30:07.231660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.231581 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea129fbd-3cf5-4889-a684-4c55c0c13c1b-tmp\") pod \"kube-auth-proxy-7c6946f9c7-gkrcg\" (UID: \"ea129fbd-3cf5-4889-a684-4c55c0c13c1b\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" Nov 02 07:30:07.231660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.231636 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea129fbd-3cf5-4889-a684-4c55c0c13c1b-tls-certs\") pod \"kube-auth-proxy-7c6946f9c7-gkrcg\" (UID: \"ea129fbd-3cf5-4889-a684-4c55c0c13c1b\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" Nov 02 07:30:07.231852 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.231730 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkt99\" (UniqueName: \"kubernetes.io/projected/ea129fbd-3cf5-4889-a684-4c55c0c13c1b-kube-api-access-wkt99\") pod \"kube-auth-proxy-7c6946f9c7-gkrcg\" (UID: \"ea129fbd-3cf5-4889-a684-4c55c0c13c1b\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" Nov 02 07:30:07.332921 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.332887 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea129fbd-3cf5-4889-a684-4c55c0c13c1b-tmp\") pod \"kube-auth-proxy-7c6946f9c7-gkrcg\" (UID: \"ea129fbd-3cf5-4889-a684-4c55c0c13c1b\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" Nov 02 07:30:07.333490 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.332947 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea129fbd-3cf5-4889-a684-4c55c0c13c1b-tls-certs\") pod \"kube-auth-proxy-7c6946f9c7-gkrcg\" (UID: \"ea129fbd-3cf5-4889-a684-4c55c0c13c1b\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" Nov 02 07:30:07.333490 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.333024 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkt99\" (UniqueName: \"kubernetes.io/projected/ea129fbd-3cf5-4889-a684-4c55c0c13c1b-kube-api-access-wkt99\") pod \"kube-auth-proxy-7c6946f9c7-gkrcg\" (UID: \"ea129fbd-3cf5-4889-a684-4c55c0c13c1b\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" Nov 02 07:30:07.335822 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.335792 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea129fbd-3cf5-4889-a684-4c55c0c13c1b-tmp\") pod \"kube-auth-proxy-7c6946f9c7-gkrcg\" (UID: \"ea129fbd-3cf5-4889-a684-4c55c0c13c1b\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" Nov 02 07:30:07.336197 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.336181 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea129fbd-3cf5-4889-a684-4c55c0c13c1b-tls-certs\") pod \"kube-auth-proxy-7c6946f9c7-gkrcg\" (UID: \"ea129fbd-3cf5-4889-a684-4c55c0c13c1b\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" Nov 02 07:30:07.357918 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.357884 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkt99\" (UniqueName: \"kubernetes.io/projected/ea129fbd-3cf5-4889-a684-4c55c0c13c1b-kube-api-access-wkt99\") pod \"kube-auth-proxy-7c6946f9c7-gkrcg\" (UID: \"ea129fbd-3cf5-4889-a684-4c55c0c13c1b\") " pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" Nov 02 07:30:07.370173 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.370130 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:07.448343 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.448305 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" Nov 02 07:30:07.519109 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.519072 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:07.665646 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.665610 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:07.748888 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:07.748860 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg"] Nov 02 07:30:07.749679 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:30:07.749629 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea129fbd_3cf5_4889_a684_4c55c0c13c1b.slice/crio-ef258441c73cdcd1d5af95aaf03a2b3a875f50b2b5e9792e54d128dfdba3c6d2 WatchSource:0}: Error finding container ef258441c73cdcd1d5af95aaf03a2b3a875f50b2b5e9792e54d128dfdba3c6d2: Status 404 returned error can't find the container with id ef258441c73cdcd1d5af95aaf03a2b3a875f50b2b5e9792e54d128dfdba3c6d2 Nov 02 07:30:08.007784 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.007711 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:08.144453 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.144414 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:08.153224 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.153192 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" event={"ID":"ea129fbd-3cf5-4889-a684-4c55c0c13c1b","Type":"ContainerStarted","Data":"ef258441c73cdcd1d5af95aaf03a2b3a875f50b2b5e9792e54d128dfdba3c6d2"} Nov 02 07:30:08.434089 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.434003 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nx647"] Nov 02 07:30:08.438285 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.438260 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" Nov 02 07:30:08.443328 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.443302 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c48ce460-8949-474f-bf8c-de701d53c794-config-file\") pod \"limitador-limitador-74dfcfb894-nx647\" (UID: \"c48ce460-8949-474f-bf8c-de701d53c794\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" Nov 02 07:30:08.443506 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.443471 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mstbn\" (UniqueName: \"kubernetes.io/projected/c48ce460-8949-474f-bf8c-de701d53c794-kube-api-access-mstbn\") pod \"limitador-limitador-74dfcfb894-nx647\" (UID: \"c48ce460-8949-474f-bf8c-de701d53c794\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" Nov 02 07:30:08.458654 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.458599 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 07:30:08.462943 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.462730 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nx647"] Nov 02 07:30:08.467875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.467842 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:08.544438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.544403 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mstbn\" (UniqueName: \"kubernetes.io/projected/c48ce460-8949-474f-bf8c-de701d53c794-kube-api-access-mstbn\") pod \"limitador-limitador-74dfcfb894-nx647\" (UID: \"c48ce460-8949-474f-bf8c-de701d53c794\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" Nov 02 07:30:08.544577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.544448 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c48ce460-8949-474f-bf8c-de701d53c794-config-file\") pod \"limitador-limitador-74dfcfb894-nx647\" (UID: \"c48ce460-8949-474f-bf8c-de701d53c794\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" Nov 02 07:30:08.545041 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.545022 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c48ce460-8949-474f-bf8c-de701d53c794-config-file\") pod \"limitador-limitador-74dfcfb894-nx647\" (UID: \"c48ce460-8949-474f-bf8c-de701d53c794\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" Nov 02 07:30:08.581510 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.581473 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nx647"] Nov 02 07:30:08.599677 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.599649 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mstbn\" (UniqueName: \"kubernetes.io/projected/c48ce460-8949-474f-bf8c-de701d53c794-kube-api-access-mstbn\") pod \"limitador-limitador-74dfcfb894-nx647\" (UID: \"c48ce460-8949-474f-bf8c-de701d53c794\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" Nov 02 07:30:08.664871 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.664820 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:08.754863 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.754827 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" Nov 02 07:30:08.802154 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:08.802123 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:09.075808 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:30:09.075766 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48ce460_8949_474f_bf8c_de701d53c794.slice/crio-9b54bbb399b914b7ec580d629276f92a11294f3ca5e45d3db78d3ebde07786a8 WatchSource:0}: Error finding container 9b54bbb399b914b7ec580d629276f92a11294f3ca5e45d3db78d3ebde07786a8: Status 404 returned error can't find the container with id 9b54bbb399b914b7ec580d629276f92a11294f3ca5e45d3db78d3ebde07786a8 Nov 02 07:30:09.075967 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:09.075947 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nx647"] Nov 02 07:30:09.160366 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:09.160328 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" event={"ID":"c48ce460-8949-474f-bf8c-de701d53c794","Type":"ContainerStarted","Data":"9b54bbb399b914b7ec580d629276f92a11294f3ca5e45d3db78d3ebde07786a8"} Nov 02 07:30:09.355546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:09.355472 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:09.593536 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:09.593432 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:09.729876 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:09.729841 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:10.531112 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:10.531077 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:10.537852 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:10.537825 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h54gn"] Nov 02 07:30:10.541691 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:10.541670 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h54gn" Nov 02 07:30:10.563291 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:10.563259 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h54gn"] Nov 02 07:30:10.564713 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:10.564688 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk2nh\" (UniqueName: \"kubernetes.io/projected/993179dd-80b3-4a52-b6e8-126997570820-kube-api-access-jk2nh\") pod \"authorino-56c6c9ddd5-h54gn\" (UID: \"993179dd-80b3-4a52-b6e8-126997570820\") " pod="kuadrant-system/authorino-56c6c9ddd5-h54gn" Nov 02 07:30:10.654815 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:10.654777 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kpq4g\"" Nov 02 07:30:10.665232 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:10.665204 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk2nh\" (UniqueName: \"kubernetes.io/projected/993179dd-80b3-4a52-b6e8-126997570820-kube-api-access-jk2nh\") pod \"authorino-56c6c9ddd5-h54gn\" (UID: \"993179dd-80b3-4a52-b6e8-126997570820\") " pod="kuadrant-system/authorino-56c6c9ddd5-h54gn" Nov 02 07:30:10.678659 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:10.678620 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:10.690394 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:10.690369 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk2nh\" (UniqueName: \"kubernetes.io/projected/993179dd-80b3-4a52-b6e8-126997570820-kube-api-access-jk2nh\") pod \"authorino-56c6c9ddd5-h54gn\" (UID: \"993179dd-80b3-4a52-b6e8-126997570820\") " pod="kuadrant-system/authorino-56c6c9ddd5-h54gn" Nov 02 07:30:10.855377 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:10.855277 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h54gn" Nov 02 07:30:11.173073 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:11.172980 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" event={"ID":"ea129fbd-3cf5-4889-a684-4c55c0c13c1b","Type":"ContainerStarted","Data":"70d7742ba31ad0364d64621662bd0026710cd9256e81a16039c08578365f5290"} Nov 02 07:30:11.189515 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:11.189472 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h54gn"] Nov 02 07:30:11.190909 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:30:11.190884 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod993179dd_80b3_4a52_b6e8_126997570820.slice/crio-5087d07b937bdc85d5dacf4db090e1172416b67e3772b62ea469ea24f13d083b WatchSource:0}: Error finding container 5087d07b937bdc85d5dacf4db090e1172416b67e3772b62ea469ea24f13d083b: Status 404 returned error can't find the container with id 5087d07b937bdc85d5dacf4db090e1172416b67e3772b62ea469ea24f13d083b Nov 02 07:30:11.230631 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:11.230575 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c6946f9c7-gkrcg" podStartSLOduration=1.7849209510000001 podStartE2EDuration="4.230555753s" totalImagesPullingTime="2.445634802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:30:11.229039054 +0000 UTC m=+311.527319895" watchObservedRunningTime="2026-11-02 07:30:11.230555753 +0000 UTC m=+311.528836580" Nov 02 07:30:12.179586 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:12.179485 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" event={"ID":"c48ce460-8949-474f-bf8c-de701d53c794","Type":"ContainerStarted","Data":"d82269b5d066f93bab5f04362d2e37af3cce9ff8756bf39a3a5cfd52f1832404"} Nov 02 07:30:12.179586 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:12.179558 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" Nov 02 07:30:12.180632 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:12.180609 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h54gn" event={"ID":"993179dd-80b3-4a52-b6e8-126997570820","Type":"ContainerStarted","Data":"5087d07b937bdc85d5dacf4db090e1172416b67e3772b62ea469ea24f13d083b"} Nov 02 07:30:12.227143 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:12.227081 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" podStartSLOduration=1.389724777 podStartE2EDuration="4.22706262s" totalImagesPullingTime="2.837337843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:30:12.223950397 +0000 UTC m=+312.522231224" watchObservedRunningTime="2026-11-02 07:30:12.22706262 +0000 UTC m=+312.525343446" Nov 02 07:30:13.140107 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:30:13.140062 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:30:14.255250 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.255213 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:14.402396 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.402354 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:14.527397 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.527259 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp"] Nov 02 07:30:14.531467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.531444 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:14.552429 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.551659 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-jsw7k\"" Nov 02 07:30:14.562126 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.562103 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 07:30:14.562247 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.562183 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 07:30:14.563414 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.563393 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp"] Nov 02 07:30:14.589790 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.589753 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 07:30:14.608774 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.608746 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dbts\" (UniqueName: \"kubernetes.io/projected/62de7c44-3318-4553-b3d0-f0124ed28ac4-kube-api-access-7dbts\") pod \"kserve-module-controller-manager-7566cf5fc8-kv7wp\" (UID: \"62de7c44-3318-4553-b3d0-f0124ed28ac4\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:14.608905 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.608788 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62de7c44-3318-4553-b3d0-f0124ed28ac4-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kv7wp\" (UID: \"62de7c44-3318-4553-b3d0-f0124ed28ac4\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:14.608905 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.608812 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62de7c44-3318-4553-b3d0-f0124ed28ac4-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kv7wp\" (UID: \"62de7c44-3318-4553-b3d0-f0124ed28ac4\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:14.609017 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.608997 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62de7c44-3318-4553-b3d0-f0124ed28ac4-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kv7wp\" (UID: \"62de7c44-3318-4553-b3d0-f0124ed28ac4\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:14.709963 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.709927 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62de7c44-3318-4553-b3d0-f0124ed28ac4-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kv7wp\" (UID: \"62de7c44-3318-4553-b3d0-f0124ed28ac4\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:14.710132 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.709971 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62de7c44-3318-4553-b3d0-f0124ed28ac4-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kv7wp\" (UID: \"62de7c44-3318-4553-b3d0-f0124ed28ac4\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:14.710132 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.710018 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62de7c44-3318-4553-b3d0-f0124ed28ac4-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kv7wp\" (UID: \"62de7c44-3318-4553-b3d0-f0124ed28ac4\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:14.710132 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.710103 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dbts\" (UniqueName: \"kubernetes.io/projected/62de7c44-3318-4553-b3d0-f0124ed28ac4-kube-api-access-7dbts\") pod \"kserve-module-controller-manager-7566cf5fc8-kv7wp\" (UID: \"62de7c44-3318-4553-b3d0-f0124ed28ac4\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:14.710448 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.710424 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62de7c44-3318-4553-b3d0-f0124ed28ac4-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kv7wp\" (UID: \"62de7c44-3318-4553-b3d0-f0124ed28ac4\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:14.710511 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.710438 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62de7c44-3318-4553-b3d0-f0124ed28ac4-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kv7wp\" (UID: \"62de7c44-3318-4553-b3d0-f0124ed28ac4\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:14.710856 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.710839 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62de7c44-3318-4553-b3d0-f0124ed28ac4-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kv7wp\" (UID: \"62de7c44-3318-4553-b3d0-f0124ed28ac4\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:14.735347 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.735322 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dbts\" (UniqueName: \"kubernetes.io/projected/62de7c44-3318-4553-b3d0-f0124ed28ac4-kube-api-access-7dbts\") pod \"kserve-module-controller-manager-7566cf5fc8-kv7wp\" (UID: \"62de7c44-3318-4553-b3d0-f0124ed28ac4\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:14.854375 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:14.854296 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:15.185341 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:15.185317 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp"] Nov 02 07:30:15.185825 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:30:15.185797 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62de7c44_3318_4553_b3d0_f0124ed28ac4.slice/crio-550adcd706e7ea2d7c002dbf82ed7d7af28a9ee19577ccb2eaeb0abeda5a558f WatchSource:0}: Error finding container 550adcd706e7ea2d7c002dbf82ed7d7af28a9ee19577ccb2eaeb0abeda5a558f: Status 404 returned error can't find the container with id 550adcd706e7ea2d7c002dbf82ed7d7af28a9ee19577ccb2eaeb0abeda5a558f Nov 02 07:30:15.197059 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:15.197026 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h54gn" event={"ID":"993179dd-80b3-4a52-b6e8-126997570820","Type":"ContainerStarted","Data":"372caab618b4801075ca9ce77a81b813d249b66c3dc967163217b4e18afe7da8"} Nov 02 07:30:15.198113 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:15.198093 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" event={"ID":"62de7c44-3318-4553-b3d0-f0124ed28ac4","Type":"ContainerStarted","Data":"550adcd706e7ea2d7c002dbf82ed7d7af28a9ee19577ccb2eaeb0abeda5a558f"} Nov 02 07:30:15.238535 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:15.238485 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-h54gn" podStartSLOduration=1.896523238 podStartE2EDuration="5.238472s" totalImagesPullingTime="3.341948762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:30:15.238152139 +0000 UTC m=+315.536432964" watchObservedRunningTime="2026-11-02 07:30:15.238472 +0000 UTC m=+315.536752824" Nov 02 07:30:17.209118 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:17.209034 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" event={"ID":"62de7c44-3318-4553-b3d0-f0124ed28ac4","Type":"ContainerStarted","Data":"56350ab2391e9a1a17560b68ff5b8410bff154534a4e1e7ecc2e46aefd660203"} Nov 02 07:30:17.209554 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:17.209175 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:17.270131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:17.270072 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" podStartSLOduration=1.669191982 podStartE2EDuration="3.270058278s" totalImagesPullingTime="1.600866296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:30:17.268928372 +0000 UTC m=+317.567209198" watchObservedRunningTime="2026-11-02 07:30:17.270058278 +0000 UTC m=+317.568339102" Nov 02 07:30:20.044674 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:20.044637 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:20.182735 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:20.182698 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:23.186042 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:23.186004 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" Nov 02 07:30:23.760459 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:23.760425 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nx647"] Nov 02 07:30:23.760660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:23.760604 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" podUID="c48ce460-8949-474f-bf8c-de701d53c794" containerName="limitador" containerID="cri-o://d82269b5d066f93bab5f04362d2e37af3cce9ff8756bf39a3a5cfd52f1832404" gracePeriod=30 Nov 02 07:30:24.242291 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:24.242249 5899 generic.go:356] "Generic (PLEG): container finished" podID="c48ce460-8949-474f-bf8c-de701d53c794" containerID="d82269b5d066f93bab5f04362d2e37af3cce9ff8756bf39a3a5cfd52f1832404" exitCode=0 Nov 02 07:30:24.242738 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:24.242338 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" event={"ID":"c48ce460-8949-474f-bf8c-de701d53c794","Type":"ContainerDied","Data":"d82269b5d066f93bab5f04362d2e37af3cce9ff8756bf39a3a5cfd52f1832404"} Nov 02 07:30:24.305480 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:24.305455 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" Nov 02 07:30:24.397202 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:24.397101 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c48ce460-8949-474f-bf8c-de701d53c794-config-file\" (UniqueName: \"kubernetes.io/configmap/c48ce460-8949-474f-bf8c-de701d53c794-config-file\") pod \"c48ce460-8949-474f-bf8c-de701d53c794\" (UID: \"c48ce460-8949-474f-bf8c-de701d53c794\") " Nov 02 07:30:24.397202 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:24.397144 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48ce460-8949-474f-bf8c-de701d53c794-kube-api-access-mstbn\" (UniqueName: \"kubernetes.io/projected/c48ce460-8949-474f-bf8c-de701d53c794-kube-api-access-mstbn\") pod \"c48ce460-8949-474f-bf8c-de701d53c794\" (UID: \"c48ce460-8949-474f-bf8c-de701d53c794\") " Nov 02 07:30:24.397514 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:24.397490 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48ce460-8949-474f-bf8c-de701d53c794-config-file" pod "c48ce460-8949-474f-bf8c-de701d53c794" (UID: "c48ce460-8949-474f-bf8c-de701d53c794"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:30:24.399360 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:24.399331 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48ce460-8949-474f-bf8c-de701d53c794-kube-api-access-mstbn" pod "c48ce460-8949-474f-bf8c-de701d53c794" (UID: "c48ce460-8949-474f-bf8c-de701d53c794"). InnerVolumeSpecName "kube-api-access-mstbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:30:24.498575 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:24.498540 5899 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c48ce460-8949-474f-bf8c-de701d53c794-config-file\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:30:24.498575 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:24.498573 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mstbn\" (UniqueName: \"kubernetes.io/projected/c48ce460-8949-474f-bf8c-de701d53c794-kube-api-access-mstbn\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:30:25.248278 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:25.248245 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" Nov 02 07:30:25.248278 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:25.248259 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nx647" event={"ID":"c48ce460-8949-474f-bf8c-de701d53c794","Type":"ContainerDied","Data":"9b54bbb399b914b7ec580d629276f92a11294f3ca5e45d3db78d3ebde07786a8"} Nov 02 07:30:25.248807 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:25.248328 5899 scope.go:122] "RemoveContainer" containerID="d82269b5d066f93bab5f04362d2e37af3cce9ff8756bf39a3a5cfd52f1832404" Nov 02 07:30:25.305177 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:25.305126 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nx647"] Nov 02 07:30:25.319067 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:25.319032 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nx647"] Nov 02 07:30:25.818596 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:25.818562 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c48ce460-8949-474f-bf8c-de701d53c794" path="/var/lib/kubelet/pods/c48ce460-8949-474f-bf8c-de701d53c794/volumes" Nov 02 07:30:29.036877 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.036842 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:29.168926 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.168892 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:29.666516 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.666483 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t"] Nov 02 07:30:29.666895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.666882 5899 cpu_manager.go:414] "containerMap: removing container" podUID="c48ce460-8949-474f-bf8c-de701d53c794" containerName="limitador" Nov 02 07:30:29.666944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.666897 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48ce460-8949-474f-bf8c-de701d53c794" containerName="limitador" Nov 02 07:30:29.667000 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.666990 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="c48ce460-8949-474f-bf8c-de701d53c794" containerName="limitador" Nov 02 07:30:29.687866 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.687836 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t" Nov 02 07:30:29.690104 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.690077 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t"] Nov 02 07:30:29.704484 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.704457 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 07:30:29.717895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.717873 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nvzwk\"" Nov 02 07:30:29.717895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.717877 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 07:30:29.718051 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.717901 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 07:30:29.744121 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.744087 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a348850-1b82-4852-9cf3-be29bbb75e6e-data\") pod \"postgres-7cdbf6dc44-mt86t\" (UID: \"1a348850-1b82-4852-9cf3-be29bbb75e6e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t" Nov 02 07:30:29.744277 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.744131 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwnr8\" (UniqueName: \"kubernetes.io/projected/1a348850-1b82-4852-9cf3-be29bbb75e6e-kube-api-access-dwnr8\") pod \"postgres-7cdbf6dc44-mt86t\" (UID: \"1a348850-1b82-4852-9cf3-be29bbb75e6e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t" Nov 02 07:30:29.844667 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.844633 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a348850-1b82-4852-9cf3-be29bbb75e6e-data\") pod \"postgres-7cdbf6dc44-mt86t\" (UID: \"1a348850-1b82-4852-9cf3-be29bbb75e6e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t" Nov 02 07:30:29.844834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.844676 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwnr8\" (UniqueName: \"kubernetes.io/projected/1a348850-1b82-4852-9cf3-be29bbb75e6e-kube-api-access-dwnr8\") pod \"postgres-7cdbf6dc44-mt86t\" (UID: \"1a348850-1b82-4852-9cf3-be29bbb75e6e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t" Nov 02 07:30:29.845082 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.845057 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a348850-1b82-4852-9cf3-be29bbb75e6e-data\") pod \"postgres-7cdbf6dc44-mt86t\" (UID: \"1a348850-1b82-4852-9cf3-be29bbb75e6e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t" Nov 02 07:30:29.854971 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.854947 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:29.869504 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:29.869474 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwnr8\" (UniqueName: \"kubernetes.io/projected/1a348850-1b82-4852-9cf3-be29bbb75e6e-kube-api-access-dwnr8\") pod \"postgres-7cdbf6dc44-mt86t\" (UID: \"1a348850-1b82-4852-9cf3-be29bbb75e6e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t" Nov 02 07:30:30.003898 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:30.003863 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t" Nov 02 07:30:30.305592 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:30.305563 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t"] Nov 02 07:30:30.306565 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:30:30.306537 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a348850_1b82_4852_9cf3_be29bbb75e6e.slice/crio-e8435af5edecd678c6213b1771e6a4e71fbd91583ed67b7808e042d7ce44aed1 WatchSource:0}: Error finding container e8435af5edecd678c6213b1771e6a4e71fbd91583ed67b7808e042d7ce44aed1: Status 404 returned error can't find the container with id e8435af5edecd678c6213b1771e6a4e71fbd91583ed67b7808e042d7ce44aed1 Nov 02 07:30:30.307922 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:30.307904 5899 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 07:30:31.287785 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:31.287742 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t" event={"ID":"1a348850-1b82-4852-9cf3-be29bbb75e6e","Type":"ContainerStarted","Data":"e8435af5edecd678c6213b1771e6a4e71fbd91583ed67b7808e042d7ce44aed1"} Nov 02 07:30:35.309939 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:35.309853 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t" event={"ID":"1a348850-1b82-4852-9cf3-be29bbb75e6e","Type":"ContainerStarted","Data":"7d9d43107caa92a563ecab397c6352411b1381cda5100135cb554520312d81b4"} Nov 02 07:30:35.309939 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:35.309929 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t" Nov 02 07:30:35.349768 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:35.349723 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t" podStartSLOduration=1.601667504 podStartE2EDuration="6.34970996s" totalImagesPullingTime="4.748042456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:30:35.349147656 +0000 UTC m=+335.647428482" watchObservedRunningTime="2026-11-02 07:30:35.34970996 +0000 UTC m=+335.647990782" Nov 02 07:30:38.214877 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:38.214842 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kv7wp" Nov 02 07:30:41.343564 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:41.343538 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mt86t" Nov 02 07:30:43.139794 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:30:43.139753 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:30:48.254839 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:48.254806 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:48.389978 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:48.389946 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:48.529139 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:48.529059 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:48.671390 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:48.671351 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:52.444584 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:52.444541 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:52.583098 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:52.583061 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:54.986438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:54.986402 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:30:55.124047 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:30:55.124018 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:00.252203 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:00.252146 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:10.005094 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.005030 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:10.151805 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.151771 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:10.253785 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.253747 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-g8nbx"] Nov 02 07:31:10.287637 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.287565 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-g8nbx"] Nov 02 07:31:10.287777 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.287670 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-g8nbx" Nov 02 07:31:10.296984 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.296954 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:10.297207 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.297187 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx"] Nov 02 07:31:10.300921 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.300902 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" Nov 02 07:31:10.319797 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.319768 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ac39273-9863-4a60-86f4-29e64cbe2946-cert\") pod \"kserve-controller-manager-776cc66d47-g8nbx\" (UID: \"5ac39273-9863-4a60-86f4-29e64cbe2946\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g8nbx" Nov 02 07:31:10.319908 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.319824 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dv9r\" (UniqueName: \"kubernetes.io/projected/5ac39273-9863-4a60-86f4-29e64cbe2946-kube-api-access-5dv9r\") pod \"kserve-controller-manager-776cc66d47-g8nbx\" (UID: \"5ac39273-9863-4a60-86f4-29e64cbe2946\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g8nbx" Nov 02 07:31:10.327452 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.327429 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx"] Nov 02 07:31:10.347087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.347061 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-pq62z"] Nov 02 07:31:10.350938 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.350916 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-pq62z" Nov 02 07:31:10.353754 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.353721 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hhhw7\"" Nov 02 07:31:10.353754 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.353724 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gshgr\"" Nov 02 07:31:10.353881 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.353751 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 07:31:10.353881 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.353806 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 07:31:10.366945 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.366928 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-mh52x\"" Nov 02 07:31:10.369535 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.369516 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 07:31:10.371270 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.371248 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-pq62z"] Nov 02 07:31:10.421536 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.421498 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ac39273-9863-4a60-86f4-29e64cbe2946-cert\") pod \"kserve-controller-manager-776cc66d47-g8nbx\" (UID: \"5ac39273-9863-4a60-86f4-29e64cbe2946\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g8nbx" Nov 02 07:31:10.421718 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.421552 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnrm9\" (UniqueName: \"kubernetes.io/projected/5cf18756-1534-40dd-9ef5-bd12f34cf968-kube-api-access-rnrm9\") pod \"llmisvc-controller-manager-869d5dcc5d-vknrx\" (UID: \"5cf18756-1534-40dd-9ef5-bd12f34cf968\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" Nov 02 07:31:10.421718 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.421590 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787dc60b-c7be-4b20-9b11-daf3140a7589-tls-certs\") pod \"model-serving-api-75bd89fdfc-pq62z\" (UID: \"787dc60b-c7be-4b20-9b11-daf3140a7589\") " pod="opendatahub/model-serving-api-75bd89fdfc-pq62z" Nov 02 07:31:10.421842 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.421778 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cf18756-1534-40dd-9ef5-bd12f34cf968-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vknrx\" (UID: \"5cf18756-1534-40dd-9ef5-bd12f34cf968\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" Nov 02 07:31:10.421842 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.421809 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw67v\" (UniqueName: \"kubernetes.io/projected/787dc60b-c7be-4b20-9b11-daf3140a7589-kube-api-access-sw67v\") pod \"model-serving-api-75bd89fdfc-pq62z\" (UID: \"787dc60b-c7be-4b20-9b11-daf3140a7589\") " pod="opendatahub/model-serving-api-75bd89fdfc-pq62z" Nov 02 07:31:10.421955 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.421842 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dv9r\" (UniqueName: \"kubernetes.io/projected/5ac39273-9863-4a60-86f4-29e64cbe2946-kube-api-access-5dv9r\") pod \"kserve-controller-manager-776cc66d47-g8nbx\" (UID: \"5ac39273-9863-4a60-86f4-29e64cbe2946\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g8nbx" Nov 02 07:31:10.424102 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.424076 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ac39273-9863-4a60-86f4-29e64cbe2946-cert\") pod \"kserve-controller-manager-776cc66d47-g8nbx\" (UID: \"5ac39273-9863-4a60-86f4-29e64cbe2946\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g8nbx" Nov 02 07:31:10.440392 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.440359 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:10.444742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.444718 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dv9r\" (UniqueName: \"kubernetes.io/projected/5ac39273-9863-4a60-86f4-29e64cbe2946-kube-api-access-5dv9r\") pod \"kserve-controller-manager-776cc66d47-g8nbx\" (UID: \"5ac39273-9863-4a60-86f4-29e64cbe2946\") " pod="opendatahub/kserve-controller-manager-776cc66d47-g8nbx" Nov 02 07:31:10.522597 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.522559 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnrm9\" (UniqueName: \"kubernetes.io/projected/5cf18756-1534-40dd-9ef5-bd12f34cf968-kube-api-access-rnrm9\") pod \"llmisvc-controller-manager-869d5dcc5d-vknrx\" (UID: \"5cf18756-1534-40dd-9ef5-bd12f34cf968\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" Nov 02 07:31:10.522597 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.522599 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787dc60b-c7be-4b20-9b11-daf3140a7589-tls-certs\") pod \"model-serving-api-75bd89fdfc-pq62z\" (UID: \"787dc60b-c7be-4b20-9b11-daf3140a7589\") " pod="opendatahub/model-serving-api-75bd89fdfc-pq62z" Nov 02 07:31:10.522838 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.522685 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cf18756-1534-40dd-9ef5-bd12f34cf968-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vknrx\" (UID: \"5cf18756-1534-40dd-9ef5-bd12f34cf968\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" Nov 02 07:31:10.522838 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.522717 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw67v\" (UniqueName: \"kubernetes.io/projected/787dc60b-c7be-4b20-9b11-daf3140a7589-kube-api-access-sw67v\") pod \"model-serving-api-75bd89fdfc-pq62z\" (UID: \"787dc60b-c7be-4b20-9b11-daf3140a7589\") " pod="opendatahub/model-serving-api-75bd89fdfc-pq62z" Nov 02 07:31:10.525348 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.525320 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787dc60b-c7be-4b20-9b11-daf3140a7589-tls-certs\") pod \"model-serving-api-75bd89fdfc-pq62z\" (UID: \"787dc60b-c7be-4b20-9b11-daf3140a7589\") " pod="opendatahub/model-serving-api-75bd89fdfc-pq62z" Nov 02 07:31:10.525538 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.525513 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cf18756-1534-40dd-9ef5-bd12f34cf968-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vknrx\" (UID: \"5cf18756-1534-40dd-9ef5-bd12f34cf968\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" Nov 02 07:31:10.545604 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.545543 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw67v\" (UniqueName: \"kubernetes.io/projected/787dc60b-c7be-4b20-9b11-daf3140a7589-kube-api-access-sw67v\") pod \"model-serving-api-75bd89fdfc-pq62z\" (UID: \"787dc60b-c7be-4b20-9b11-daf3140a7589\") " pod="opendatahub/model-serving-api-75bd89fdfc-pq62z" Nov 02 07:31:10.545708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.545683 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnrm9\" (UniqueName: \"kubernetes.io/projected/5cf18756-1534-40dd-9ef5-bd12f34cf968-kube-api-access-rnrm9\") pod \"llmisvc-controller-manager-869d5dcc5d-vknrx\" (UID: \"5cf18756-1534-40dd-9ef5-bd12f34cf968\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" Nov 02 07:31:10.569447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.569396 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:10.602115 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.602079 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-g8nbx" Nov 02 07:31:10.612368 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.612231 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" Nov 02 07:31:10.665023 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.664721 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-pq62z" Nov 02 07:31:10.806000 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.805898 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:10.915835 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.915655 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-g8nbx"] Nov 02 07:31:10.921988 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.921886 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx"] Nov 02 07:31:10.981574 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:10.981538 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-pq62z"] Nov 02 07:31:10.983696 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:31:10.983658 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787dc60b_c7be_4b20_9b11_daf3140a7589.slice/crio-78f33fc99e4277b6b237291a7cc4ee0f3a033ec08b69015aa53b54d1100eb5a5 WatchSource:0}: Error finding container 78f33fc99e4277b6b237291a7cc4ee0f3a033ec08b69015aa53b54d1100eb5a5: Status 404 returned error can't find the container with id 78f33fc99e4277b6b237291a7cc4ee0f3a033ec08b69015aa53b54d1100eb5a5 Nov 02 07:31:11.485823 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:11.485783 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-g8nbx" event={"ID":"5ac39273-9863-4a60-86f4-29e64cbe2946","Type":"ContainerStarted","Data":"2f7f5d2ee1c31277c961b9ccc0a050dd6308ba97e13251d628168cc6188486cc"} Nov 02 07:31:11.487316 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:11.487278 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-pq62z" event={"ID":"787dc60b-c7be-4b20-9b11-daf3140a7589","Type":"ContainerStarted","Data":"78f33fc99e4277b6b237291a7cc4ee0f3a033ec08b69015aa53b54d1100eb5a5"} Nov 02 07:31:11.488761 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:11.488723 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" event={"ID":"5cf18756-1534-40dd-9ef5-bd12f34cf968","Type":"ContainerStarted","Data":"8f979f3f71018e87cc628540dd610853a4f9e880a506d421ac7bf0efdb55e3b9"} Nov 02 07:31:12.683736 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:12.683674 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:12.822076 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:12.822039 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:12.963611 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:12.963582 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:13.096443 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:13.096401 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:13.147785 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:31:13.147741 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:31:13.500708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:13.500668 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" event={"ID":"5cf18756-1534-40dd-9ef5-bd12f34cf968","Type":"ContainerStarted","Data":"bacb8707907c66f2e2484adc910ea8a6a24cfbbbf30840a9fa13c4dc50b1a290"} Nov 02 07:31:13.500882 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:13.500743 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" Nov 02 07:31:13.502396 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:13.502371 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-g8nbx" event={"ID":"5ac39273-9863-4a60-86f4-29e64cbe2946","Type":"ContainerStarted","Data":"9da2c4deee424f927fdabc3475a9d966f700250ca314126ffe0642bd5f57a6b6"} Nov 02 07:31:13.502539 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:13.502441 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-g8nbx" Nov 02 07:31:13.503944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:13.503923 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-pq62z" event={"ID":"787dc60b-c7be-4b20-9b11-daf3140a7589","Type":"ContainerStarted","Data":"94bc66df26fa3e728f6e3df76d841e8bba870acdff07eb17d957ba5a596af381"} Nov 02 07:31:13.504052 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:13.504042 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-pq62z" Nov 02 07:31:13.540878 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:13.540772 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" podStartSLOduration=1.196240252 podStartE2EDuration="3.540757702s" totalImagesPullingTime="2.34451745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:31:13.540439211 +0000 UTC m=+373.838720037" watchObservedRunningTime="2026-11-02 07:31:13.540757702 +0000 UTC m=+373.839038559" Nov 02 07:31:13.580675 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:13.580624 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-pq62z" podStartSLOduration=1.6237372799999998 podStartE2EDuration="3.580607979s" totalImagesPullingTime="1.956870699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:31:13.580334305 +0000 UTC m=+373.878615143" watchObservedRunningTime="2026-11-02 07:31:13.580607979 +0000 UTC m=+373.878888807" Nov 02 07:31:13.619999 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:13.619948 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-g8nbx" podStartSLOduration=1.1525610099999999 podStartE2EDuration="3.619936609s" totalImagesPullingTime="2.467375599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:31:13.619590538 +0000 UTC m=+373.917871364" watchObservedRunningTime="2026-11-02 07:31:13.619936609 +0000 UTC m=+373.918217431" Nov 02 07:31:24.269775 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:24.269720 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:24.407181 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:24.407127 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:24.512284 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:24.512253 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-pq62z" Nov 02 07:31:25.753944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:25.753914 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:25.893356 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:25.893323 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:26.032134 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:26.032043 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:26.168228 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:26.168198 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:35.599953 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:35.599914 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:35.741293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:35.741241 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:31:37.773649 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:37.773618 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h54gn"] Nov 02 07:31:37.774305 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:37.773852 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-h54gn" podUID="993179dd-80b3-4a52-b6e8-126997570820" containerName="authorino" containerID="cri-o://372caab618b4801075ca9ce77a81b813d249b66c3dc967163217b4e18afe7da8" gracePeriod=30 Nov 02 07:31:38.027753 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:38.027693 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h54gn" Nov 02 07:31:38.175379 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:38.175345 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993179dd-80b3-4a52-b6e8-126997570820-kube-api-access-jk2nh\" (UniqueName: \"kubernetes.io/projected/993179dd-80b3-4a52-b6e8-126997570820-kube-api-access-jk2nh\") pod \"993179dd-80b3-4a52-b6e8-126997570820\" (UID: \"993179dd-80b3-4a52-b6e8-126997570820\") " Nov 02 07:31:38.177630 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:38.177597 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993179dd-80b3-4a52-b6e8-126997570820-kube-api-access-jk2nh" pod "993179dd-80b3-4a52-b6e8-126997570820" (UID: "993179dd-80b3-4a52-b6e8-126997570820"). InnerVolumeSpecName "kube-api-access-jk2nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:31:38.276321 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:38.276284 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk2nh\" (UniqueName: \"kubernetes.io/projected/993179dd-80b3-4a52-b6e8-126997570820-kube-api-access-jk2nh\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:31:38.620257 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:38.620224 5899 generic.go:356] "Generic (PLEG): container finished" podID="993179dd-80b3-4a52-b6e8-126997570820" containerID="372caab618b4801075ca9ce77a81b813d249b66c3dc967163217b4e18afe7da8" exitCode=0 Nov 02 07:31:38.620442 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:38.620270 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h54gn" Nov 02 07:31:38.620442 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:38.620313 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h54gn" event={"ID":"993179dd-80b3-4a52-b6e8-126997570820","Type":"ContainerDied","Data":"372caab618b4801075ca9ce77a81b813d249b66c3dc967163217b4e18afe7da8"} Nov 02 07:31:38.620442 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:38.620356 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h54gn" event={"ID":"993179dd-80b3-4a52-b6e8-126997570820","Type":"ContainerDied","Data":"5087d07b937bdc85d5dacf4db090e1172416b67e3772b62ea469ea24f13d083b"} Nov 02 07:31:38.620442 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:38.620372 5899 scope.go:122] "RemoveContainer" containerID="372caab618b4801075ca9ce77a81b813d249b66c3dc967163217b4e18afe7da8" Nov 02 07:31:38.629132 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:38.629110 5899 scope.go:122] "RemoveContainer" containerID="372caab618b4801075ca9ce77a81b813d249b66c3dc967163217b4e18afe7da8" Nov 02 07:31:38.629438 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:31:38.629421 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372caab618b4801075ca9ce77a81b813d249b66c3dc967163217b4e18afe7da8\": container with ID starting with 372caab618b4801075ca9ce77a81b813d249b66c3dc967163217b4e18afe7da8 not found: ID does not exist" containerID="372caab618b4801075ca9ce77a81b813d249b66c3dc967163217b4e18afe7da8" Nov 02 07:31:38.629501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:38.629451 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372caab618b4801075ca9ce77a81b813d249b66c3dc967163217b4e18afe7da8"} err="failed to get container status \"372caab618b4801075ca9ce77a81b813d249b66c3dc967163217b4e18afe7da8\": rpc error: code = NotFound desc = could not find container \"372caab618b4801075ca9ce77a81b813d249b66c3dc967163217b4e18afe7da8\": container with ID starting with 372caab618b4801075ca9ce77a81b813d249b66c3dc967163217b4e18afe7da8 not found: ID does not exist" Nov 02 07:31:38.668217 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:38.668186 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h54gn"] Nov 02 07:31:38.679931 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:38.679900 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h54gn"] Nov 02 07:31:39.818424 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:39.818391 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993179dd-80b3-4a52-b6e8-126997570820" path="/var/lib/kubelet/pods/993179dd-80b3-4a52-b6e8-126997570820/volumes" Nov 02 07:31:43.139994 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:31:43.139945 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:31:44.510554 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:44.510521 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" Nov 02 07:31:44.513805 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:31:44.513785 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-g8nbx" Nov 02 07:32:00.369038 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:00.368964 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:32:13.139698 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:32:13.139655 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:32:19.023406 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.023367 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-kqv85"] Nov 02 07:32:19.023824 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.023780 5899 cpu_manager.go:414] "containerMap: removing container" podUID="993179dd-80b3-4a52-b6e8-126997570820" containerName="authorino" Nov 02 07:32:19.023824 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.023792 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993179dd-80b3-4a52-b6e8-126997570820" containerName="authorino" Nov 02 07:32:19.023898 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.023852 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="993179dd-80b3-4a52-b6e8-126997570820" containerName="authorino" Nov 02 07:32:19.026585 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.026562 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kqv85" Nov 02 07:32:19.048477 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.048441 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-kqv85"] Nov 02 07:32:19.157828 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.157787 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbss\" (UniqueName: \"kubernetes.io/projected/580bc81e-1b83-472a-9af3-153ba348eab7-kube-api-access-kxbss\") pod \"llmisvc-controller-manager-6466c554d8-kqv85\" (UID: \"580bc81e-1b83-472a-9af3-153ba348eab7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-kqv85" Nov 02 07:32:19.157984 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.157846 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/580bc81e-1b83-472a-9af3-153ba348eab7-cert\") pod \"llmisvc-controller-manager-6466c554d8-kqv85\" (UID: \"580bc81e-1b83-472a-9af3-153ba348eab7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-kqv85" Nov 02 07:32:19.259206 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.259137 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbss\" (UniqueName: \"kubernetes.io/projected/580bc81e-1b83-472a-9af3-153ba348eab7-kube-api-access-kxbss\") pod \"llmisvc-controller-manager-6466c554d8-kqv85\" (UID: \"580bc81e-1b83-472a-9af3-153ba348eab7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-kqv85" Nov 02 07:32:19.259349 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.259294 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/580bc81e-1b83-472a-9af3-153ba348eab7-cert\") pod \"llmisvc-controller-manager-6466c554d8-kqv85\" (UID: \"580bc81e-1b83-472a-9af3-153ba348eab7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-kqv85" Nov 02 07:32:19.261718 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.261691 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/580bc81e-1b83-472a-9af3-153ba348eab7-cert\") pod \"llmisvc-controller-manager-6466c554d8-kqv85\" (UID: \"580bc81e-1b83-472a-9af3-153ba348eab7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-kqv85" Nov 02 07:32:19.282315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.282253 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbss\" (UniqueName: \"kubernetes.io/projected/580bc81e-1b83-472a-9af3-153ba348eab7-kube-api-access-kxbss\") pod \"llmisvc-controller-manager-6466c554d8-kqv85\" (UID: \"580bc81e-1b83-472a-9af3-153ba348eab7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-kqv85" Nov 02 07:32:19.339958 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.339906 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kqv85" Nov 02 07:32:19.634876 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.634730 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-kqv85"] Nov 02 07:32:19.637431 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:32:19.637402 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580bc81e_1b83_472a_9af3_153ba348eab7.slice/crio-19ab213a5c1f6940ab0a1913b7620d69c1db804f8e1defe1d3a39073e02f5d63 WatchSource:0}: Error finding container 19ab213a5c1f6940ab0a1913b7620d69c1db804f8e1defe1d3a39073e02f5d63: Status 404 returned error can't find the container with id 19ab213a5c1f6940ab0a1913b7620d69c1db804f8e1defe1d3a39073e02f5d63 Nov 02 07:32:19.821015 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:19.820980 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kqv85" event={"ID":"580bc81e-1b83-472a-9af3-153ba348eab7","Type":"ContainerStarted","Data":"19ab213a5c1f6940ab0a1913b7620d69c1db804f8e1defe1d3a39073e02f5d63"} Nov 02 07:32:20.828007 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:20.827969 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kqv85" event={"ID":"580bc81e-1b83-472a-9af3-153ba348eab7","Type":"ContainerStarted","Data":"e3416bcbe5e571190f0e242ffd60d4ab8a8da2a6ee7a64e1109e50e6a3eda6d8"} Nov 02 07:32:20.828407 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:20.828182 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kqv85" Nov 02 07:32:20.867478 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:20.867425 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kqv85" podStartSLOduration=1.529917124 podStartE2EDuration="1.867411768s" totalImagesPullingTime="337.494644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:32:20.867254865 +0000 UTC m=+441.165535689" watchObservedRunningTime="2026-11-02 07:32:20.867411768 +0000 UTC m=+441.165692591" Nov 02 07:32:43.140022 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:32:43.139939 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:32:51.834695 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:51.834664 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kqv85" Nov 02 07:32:51.912413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:51.912377 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx"] Nov 02 07:32:51.970069 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:51.970032 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" podUID="5cf18756-1534-40dd-9ef5-bd12f34cf968" containerName="manager" containerID="cri-o://bacb8707907c66f2e2484adc910ea8a6a24cfbbbf30840a9fa13c4dc50b1a290" gracePeriod=30 Nov 02 07:32:52.206698 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.206674 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" Nov 02 07:32:52.265188 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.265143 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf18756-1534-40dd-9ef5-bd12f34cf968-kube-api-access-rnrm9\" (UniqueName: \"kubernetes.io/projected/5cf18756-1534-40dd-9ef5-bd12f34cf968-kube-api-access-rnrm9\") pod \"5cf18756-1534-40dd-9ef5-bd12f34cf968\" (UID: \"5cf18756-1534-40dd-9ef5-bd12f34cf968\") " Nov 02 07:32:52.265362 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.265305 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cf18756-1534-40dd-9ef5-bd12f34cf968-cert\" (UniqueName: \"kubernetes.io/secret/5cf18756-1534-40dd-9ef5-bd12f34cf968-cert\") pod \"5cf18756-1534-40dd-9ef5-bd12f34cf968\" (UID: \"5cf18756-1534-40dd-9ef5-bd12f34cf968\") " Nov 02 07:32:52.267677 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.267548 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf18756-1534-40dd-9ef5-bd12f34cf968-kube-api-access-rnrm9" pod "5cf18756-1534-40dd-9ef5-bd12f34cf968" (UID: "5cf18756-1534-40dd-9ef5-bd12f34cf968"). InnerVolumeSpecName "kube-api-access-rnrm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:32:52.267677 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.267581 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf18756-1534-40dd-9ef5-bd12f34cf968-cert" pod "5cf18756-1534-40dd-9ef5-bd12f34cf968" (UID: "5cf18756-1534-40dd-9ef5-bd12f34cf968"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:32:52.366613 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.366539 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnrm9\" (UniqueName: \"kubernetes.io/projected/5cf18756-1534-40dd-9ef5-bd12f34cf968-kube-api-access-rnrm9\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:32:52.366613 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.366566 5899 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cf18756-1534-40dd-9ef5-bd12f34cf968-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:32:52.983094 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.983041 5899 generic.go:356] "Generic (PLEG): container finished" podID="5cf18756-1534-40dd-9ef5-bd12f34cf968" containerID="bacb8707907c66f2e2484adc910ea8a6a24cfbbbf30840a9fa13c4dc50b1a290" exitCode=0 Nov 02 07:32:52.984081 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.983112 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" event={"ID":"5cf18756-1534-40dd-9ef5-bd12f34cf968","Type":"ContainerDied","Data":"bacb8707907c66f2e2484adc910ea8a6a24cfbbbf30840a9fa13c4dc50b1a290"} Nov 02 07:32:52.984081 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.983180 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" event={"ID":"5cf18756-1534-40dd-9ef5-bd12f34cf968","Type":"ContainerDied","Data":"8f979f3f71018e87cc628540dd610853a4f9e880a506d421ac7bf0efdb55e3b9"} Nov 02 07:32:52.984081 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.983213 5899 scope.go:122] "RemoveContainer" containerID="bacb8707907c66f2e2484adc910ea8a6a24cfbbbf30840a9fa13c4dc50b1a290" Nov 02 07:32:52.984081 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.983373 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx" Nov 02 07:32:52.994205 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.994181 5899 scope.go:122] "RemoveContainer" containerID="bacb8707907c66f2e2484adc910ea8a6a24cfbbbf30840a9fa13c4dc50b1a290" Nov 02 07:32:52.994473 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:32:52.994453 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bacb8707907c66f2e2484adc910ea8a6a24cfbbbf30840a9fa13c4dc50b1a290\": container with ID starting with bacb8707907c66f2e2484adc910ea8a6a24cfbbbf30840a9fa13c4dc50b1a290 not found: ID does not exist" containerID="bacb8707907c66f2e2484adc910ea8a6a24cfbbbf30840a9fa13c4dc50b1a290" Nov 02 07:32:52.994567 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:52.994480 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bacb8707907c66f2e2484adc910ea8a6a24cfbbbf30840a9fa13c4dc50b1a290"} err="failed to get container status \"bacb8707907c66f2e2484adc910ea8a6a24cfbbbf30840a9fa13c4dc50b1a290\": rpc error: code = NotFound desc = could not find container \"bacb8707907c66f2e2484adc910ea8a6a24cfbbbf30840a9fa13c4dc50b1a290\": container with ID starting with bacb8707907c66f2e2484adc910ea8a6a24cfbbbf30840a9fa13c4dc50b1a290 not found: ID does not exist" Nov 02 07:32:53.038456 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:53.038394 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx"] Nov 02 07:32:53.051222 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:53.051189 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vknrx"] Nov 02 07:32:53.052809 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:53.052785 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:32:53.286511 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:53.286433 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:32:53.818140 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:53.818105 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf18756-1534-40dd-9ef5-bd12f34cf968" path="/var/lib/kubelet/pods/5cf18756-1534-40dd-9ef5-bd12f34cf968/volumes" Nov 02 07:32:58.122964 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:58.122929 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:32:58.277460 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:32:58.277427 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:00.486271 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:00.486239 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:13.142449 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:33:13.142410 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:33:22.074011 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:22.073973 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:22.549900 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:22.549866 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:23.722119 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:23.722087 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:23.900498 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:23.900465 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:27.674002 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:27.673971 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:29.179930 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:29.179892 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:32.944594 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:32.944562 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:36.477111 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:36.477078 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:37.746007 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:37.745974 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:38.301226 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:38.301192 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:38.444925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:38.444891 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:40.079335 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:40.079240 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:43.139932 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:33:43.139894 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:33:43.810833 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:43.810800 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:45.456737 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:45.456708 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:33:46.439557 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:33:46.439520 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:00.611342 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:00.611308 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:02.522711 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.522678 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc"] Nov 02 07:34:02.523221 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.523202 5899 cpu_manager.go:414] "containerMap: removing container" podUID="5cf18756-1534-40dd-9ef5-bd12f34cf968" containerName="manager" Nov 02 07:34:02.523270 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.523225 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf18756-1534-40dd-9ef5-bd12f34cf968" containerName="manager" Nov 02 07:34:02.523370 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.523355 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf18756-1534-40dd-9ef5-bd12f34cf968" containerName="manager" Nov 02 07:34:02.526677 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.526657 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.548651 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.548623 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 07:34:02.561829 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.561803 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc"] Nov 02 07:34:02.628920 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.628887 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7cxdc\"" Nov 02 07:34:02.709643 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.709611 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.709828 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.709658 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.709828 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.709679 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.709828 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.709813 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwxpw\" (UniqueName: \"kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-kube-api-access-zwxpw\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.709955 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.709843 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.709955 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.709905 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.709955 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.709928 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.710074 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.709986 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.710074 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.710018 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.811311 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.811221 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwxpw\" (UniqueName: \"kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-kube-api-access-zwxpw\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.811311 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.811261 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.811311 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.811292 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.811311 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.811308 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.811683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.811342 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.811683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.811373 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.811683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.811405 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.811683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.811452 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.811683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.811478 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.811904 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.811705 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.811904 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.811788 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.811904 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.811850 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.811904 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.811875 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.812155 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.812128 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.813754 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.813733 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.813900 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.813882 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.833964 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.833930 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.834180 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.834138 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwxpw\" (UniqueName: \"kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-kube-api-access-zwxpw\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cljsc\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:02.840478 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:02.840459 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:03.153450 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:03.153424 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc"] Nov 02 07:34:03.155944 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:34:03.155898 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e41c1b1_9b63_4bc3_b199_0d174a0cc8a5.slice/crio-a37f722b1e783b246459859c5e3b63d9c606593ce3a030449a1fb03f9f7aae0a WatchSource:0}: Error finding container a37f722b1e783b246459859c5e3b63d9c606593ce3a030449a1fb03f9f7aae0a: Status 404 returned error can't find the container with id a37f722b1e783b246459859c5e3b63d9c606593ce3a030449a1fb03f9f7aae0a Nov 02 07:34:03.319402 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:03.319364 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" event={"ID":"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5","Type":"ContainerStarted","Data":"a37f722b1e783b246459859c5e3b63d9c606593ce3a030449a1fb03f9f7aae0a"} Nov 02 07:34:05.110559 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:05.110521 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:05.252657 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:05.252622 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:05.963717 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:05.963685 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:34:05.963823 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:05.963773 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:34:05.963823 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:05.963801 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:34:06.337515 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:06.337474 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" event={"ID":"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5","Type":"ContainerStarted","Data":"ea83ba6df2c7237f1aa122d163d80c5901b597a1db819994d874c2ea42e04cf0"} Nov 02 07:34:06.409051 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:06.408994 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" podStartSLOduration=1.60393966 podStartE2EDuration="4.408977853s" totalImagesPullingTime="2.805038193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:06.396674241 +0000 UTC m=+546.694955090" watchObservedRunningTime="2026-11-02 07:34:06.408977853 +0000 UTC m=+546.707258680" Nov 02 07:34:06.605245 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:06.605146 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:06.780767 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:06.780727 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:06.841226 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:06.841196 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:07.843025 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:07.842992 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cljsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 07:34:07.843432 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:07.843056 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" podUID="9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 07:34:08.841110 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:08.841072 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cljsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 07:34:08.841308 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:08.841126 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" podUID="9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 07:34:09.841194 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:09.841147 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cljsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 07:34:09.841582 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:09.841220 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" podUID="9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 07:34:10.841758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:10.841673 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cljsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 07:34:10.841758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:10.841730 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" podUID="9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 07:34:11.841671 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:11.841639 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cljsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 07:34:11.841847 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:11.841691 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" podUID="9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 07:34:12.841210 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:12.841176 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cljsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 07:34:12.841377 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:12.841227 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" podUID="9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 07:34:13.141318 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:34:13.141233 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:34:14.181285 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:14.181249 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:14.182269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:14.182196 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:14.183001 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:14.182981 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:34:17.556791 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.556754 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7"] Nov 02 07:34:17.560912 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.560886 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.586504 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.586470 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7"] Nov 02 07:34:17.680070 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.680036 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbldj\" (UniqueName: \"kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-kube-api-access-wbldj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.680292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.680075 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.680292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.680097 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.680292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.680200 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.680292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.680259 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.680466 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.680345 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.680466 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.680388 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.680549 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.680515 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.680591 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.680552 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.781751 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.781717 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbldj\" (UniqueName: \"kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-kube-api-access-wbldj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.781751 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.781754 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.782010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.781772 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.782010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.781796 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.782010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.781812 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.782010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.781849 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.782010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.781889 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.782010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.781931 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.782010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.781956 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.782336 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.782206 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.782401 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.782344 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.782401 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.782378 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.782491 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.782458 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.782581 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.782562 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.784188 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.784151 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.784443 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.784425 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.804486 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.804464 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.804611 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.804526 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbldj\" (UniqueName: \"kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-kube-api-access-wbldj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-d2sj7\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:17.875185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:17.875095 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:18.189941 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:18.189907 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7"] Nov 02 07:34:18.192573 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:34:18.192544 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3498375e_3b5a_4be9_92b1_dcfb2ebd76c5.slice/crio-eb8d6f1802a49254e93845f25e9e9340414890668510cfa619287968d47c5acd WatchSource:0}: Error finding container eb8d6f1802a49254e93845f25e9e9340414890668510cfa619287968d47c5acd: Status 404 returned error can't find the container with id eb8d6f1802a49254e93845f25e9e9340414890668510cfa619287968d47c5acd Nov 02 07:34:18.195752 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:18.195722 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:34:18.195823 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:18.195789 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:34:18.195823 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:18.195816 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:34:18.420232 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:18.420190 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" event={"ID":"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5","Type":"ContainerStarted","Data":"356b48937f9318be9aa0388bf6388807344fcdf085c26f3aac3248e42ed3f895"} Nov 02 07:34:18.420399 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:18.420252 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" event={"ID":"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5","Type":"ContainerStarted","Data":"eb8d6f1802a49254e93845f25e9e9340414890668510cfa619287968d47c5acd"} Nov 02 07:34:18.464131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:18.464035 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" podStartSLOduration=1.464020507 podStartE2EDuration="1.464020507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:18.463577748 +0000 UTC m=+558.761858587" watchObservedRunningTime="2026-11-02 07:34:18.464020507 +0000 UTC m=+558.762301331" Nov 02 07:34:18.878184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:18.877245 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:19.877033 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:19.876992 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-d2sj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 07:34:19.880285 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:19.880251 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" podUID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 07:34:19.881492 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:19.881465 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:20.015016 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:20.014982 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:20.880546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:20.880508 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-d2sj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 07:34:20.880930 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:20.880574 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" podUID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 07:34:21.876185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:21.875585 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-d2sj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 07:34:21.876185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:21.875653 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" podUID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 07:34:22.878595 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:22.878552 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-d2sj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 07:34:22.879236 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:22.879199 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" podUID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 07:34:23.880189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:23.877545 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-d2sj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 07:34:23.880189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:23.877607 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" podUID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 07:34:24.875715 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:24.875680 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-d2sj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 07:34:24.875885 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:24.875737 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" podUID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 07:34:26.372893 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:26.372853 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:26.373390 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:26.373263 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:26.374128 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:26.374106 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:34:30.366323 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:30.366292 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:30.513570 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:30.513536 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:30.651119 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:30.651042 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:30.800585 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:30.800549 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:31.154848 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:31.154816 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:31.319890 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:31.319861 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:31.461416 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:31.461337 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:31.595308 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:31.595273 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:31.749875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:31.749841 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:31.887535 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:31.887502 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:32.435111 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:32.435076 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:32.897080 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:32.897045 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:33.050467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:33.050427 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:34.505857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:34.505827 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:34.755209 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:34.755174 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:34.899926 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:34.899829 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:35.077644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:35.077609 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:35.221476 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:35.221384 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:35.953395 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:35.953351 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:36.092024 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:36.091987 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:37.661567 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:37.661488 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:37.929795 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:37.929711 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:38.097877 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:38.097821 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:38.248036 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:38.248000 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:38.390307 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:38.390273 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:38.527042 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:38.526960 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:38.756680 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:38.756649 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:38.898296 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:38.898220 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:38.991218 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:38.991176 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7"] Nov 02 07:34:38.998829 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:38.998792 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" Nov 02 07:34:39.041255 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.041223 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7"] Nov 02 07:34:39.051576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.051540 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-be1fb7c9\"" Nov 02 07:34:39.051576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.051542 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-be1fb7c9-dockercfg-6s8lv\"" Nov 02 07:34:39.092737 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.092696 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32f56ab-c29b-4490-a610-39d5b95253e9-plugins-config-volume\") pod \"payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7\" (UID: \"b32f56ab-c29b-4490-a610-39d5b95253e9\") " pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" Nov 02 07:34:39.092737 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.092742 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmxlt\" (UniqueName: \"kubernetes.io/projected/b32f56ab-c29b-4490-a610-39d5b95253e9-kube-api-access-wmxlt\") pod \"payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7\" (UID: \"b32f56ab-c29b-4490-a610-39d5b95253e9\") " pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" Nov 02 07:34:39.194207 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.194101 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32f56ab-c29b-4490-a610-39d5b95253e9-plugins-config-volume\") pod \"payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7\" (UID: \"b32f56ab-c29b-4490-a610-39d5b95253e9\") " pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" Nov 02 07:34:39.194492 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.194465 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmxlt\" (UniqueName: \"kubernetes.io/projected/b32f56ab-c29b-4490-a610-39d5b95253e9-kube-api-access-wmxlt\") pod \"payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7\" (UID: \"b32f56ab-c29b-4490-a610-39d5b95253e9\") " pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" Nov 02 07:34:39.196434 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.196406 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32f56ab-c29b-4490-a610-39d5b95253e9-plugins-config-volume\") pod \"payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7\" (UID: \"b32f56ab-c29b-4490-a610-39d5b95253e9\") " pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" Nov 02 07:34:39.218405 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.218376 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmxlt\" (UniqueName: \"kubernetes.io/projected/b32f56ab-c29b-4490-a610-39d5b95253e9-kube-api-access-wmxlt\") pod \"payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7\" (UID: \"b32f56ab-c29b-4490-a610-39d5b95253e9\") " pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" Nov 02 07:34:39.243155 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.243113 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:39.317112 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.317074 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" Nov 02 07:34:39.385014 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.384939 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:39.616987 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.616722 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7"] Nov 02 07:34:39.619173 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:34:39.619125 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32f56ab_c29b_4490_a610_39d5b95253e9.slice/crio-a51fb1a377a0f4e61d9cb58c61eddfad566a841fc1c357c35a32e44873b8ae18 WatchSource:0}: Error finding container a51fb1a377a0f4e61d9cb58c61eddfad566a841fc1c357c35a32e44873b8ae18: Status 404 returned error can't find the container with id a51fb1a377a0f4e61d9cb58c61eddfad566a841fc1c357c35a32e44873b8ae18 Nov 02 07:34:39.869303 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:39.869212 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:40.006234 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:40.006197 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:40.138869 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:40.138786 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:40.525402 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:40.525367 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" event={"ID":"b32f56ab-c29b-4490-a610-39d5b95253e9","Type":"ContainerStarted","Data":"a51fb1a377a0f4e61d9cb58c61eddfad566a841fc1c357c35a32e44873b8ae18"} Nov 02 07:34:41.165405 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.165368 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:41.303871 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.303841 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:41.403520 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.403487 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg"] Nov 02 07:34:41.407467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.407442 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" Nov 02 07:34:41.430937 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.430884 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg"] Nov 02 07:34:41.438199 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.438172 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8ff8e809\"" Nov 02 07:34:41.438199 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.438174 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8ff8e809-dockercfg-nhd8t\"" Nov 02 07:34:41.517470 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.517432 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-plugins-config-volume\") pod \"payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg\" (UID: \"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209\") " pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" Nov 02 07:34:41.517653 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.517556 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qdq\" (UniqueName: \"kubernetes.io/projected/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-kube-api-access-w7qdq\") pod \"payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg\" (UID: \"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209\") " pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" Nov 02 07:34:41.618766 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.618735 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-plugins-config-volume\") pod \"payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg\" (UID: \"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209\") " pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" Nov 02 07:34:41.618925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.618828 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qdq\" (UniqueName: \"kubernetes.io/projected/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-kube-api-access-w7qdq\") pod \"payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg\" (UID: \"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209\") " pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" Nov 02 07:34:41.619469 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.619448 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-plugins-config-volume\") pod \"payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg\" (UID: \"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209\") " pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" Nov 02 07:34:41.641782 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.641756 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qdq\" (UniqueName: \"kubernetes.io/projected/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-kube-api-access-w7qdq\") pod \"payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg\" (UID: \"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209\") " pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" Nov 02 07:34:41.724117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.724040 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" Nov 02 07:34:41.924644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:41.924603 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:42.022523 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:42.022500 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg"] Nov 02 07:34:42.025430 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:34:42.025400 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4374b0_f5ce_46d8_ac0a_6442a1bf7209.slice/crio-4f5f64fe6085c4a35a3812511fe0e1b98a165bbc49a23e33cfa8b0a4f8d85aeb WatchSource:0}: Error finding container 4f5f64fe6085c4a35a3812511fe0e1b98a165bbc49a23e33cfa8b0a4f8d85aeb: Status 404 returned error can't find the container with id 4f5f64fe6085c4a35a3812511fe0e1b98a165bbc49a23e33cfa8b0a4f8d85aeb Nov 02 07:34:42.537428 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:42.537391 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" event={"ID":"b32f56ab-c29b-4490-a610-39d5b95253e9","Type":"ContainerStarted","Data":"3a9d1bf9797d00a63ae7cd68beb0a141e2374ea82b87df4e7e27a6f5f204ddf5"} Nov 02 07:34:42.537903 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:42.537544 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" Nov 02 07:34:42.539078 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:42.539056 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" event={"ID":"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209","Type":"ContainerStarted","Data":"f7f68a61dd38242c538a25d91c41a8623f595231ab1d6573d4f33c90a9801533"} Nov 02 07:34:42.539202 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:42.539095 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" event={"ID":"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209","Type":"ContainerStarted","Data":"4f5f64fe6085c4a35a3812511fe0e1b98a165bbc49a23e33cfa8b0a4f8d85aeb"} Nov 02 07:34:42.539202 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:42.539123 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" Nov 02 07:34:42.565499 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:42.565459 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:42.582401 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:42.582349 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" podStartSLOduration=2.603302723 podStartE2EDuration="4.582335739s" totalImagesPullingTime="1.979033016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:42.581506168 +0000 UTC m=+582.879786993" watchObservedRunningTime="2026-11-02 07:34:42.582335739 +0000 UTC m=+582.880616562" Nov 02 07:34:42.625932 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:42.625882 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" podStartSLOduration=1.625869121 podStartE2EDuration="1.625869121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:42.625591164 +0000 UTC m=+582.923871989" watchObservedRunningTime="2026-11-02 07:34:42.625869121 +0000 UTC m=+582.924149945" Nov 02 07:34:42.805543 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:42.805457 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:43.139666 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:34:43.139584 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:34:44.033342 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.033306 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq"] Nov 02 07:34:44.036712 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.036692 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.051279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.051260 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-935d4e21-serving-cert\"" Nov 02 07:34:44.063011 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.062984 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4tgq8\"" Nov 02 07:34:44.063152 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.063134 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 07:34:44.065233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.065211 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq"] Nov 02 07:34:44.143436 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.143394 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-maas-api-tls\") pod \"maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.143436 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.143439 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e426fb-8668-4ca2-b86c-63638c21624f-openshift-service-ca\") pod \"maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.143660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.143542 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5wtw\" (UniqueName: \"kubernetes.io/projected/b3e426fb-8668-4ca2-b86c-63638c21624f-kube-api-access-s5wtw\") pod \"maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.143660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.143585 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-metrics-certs\") pod \"maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.160137 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.160109 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:44.244994 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.244963 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-maas-api-tls\") pod \"maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.244994 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.245000 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e426fb-8668-4ca2-b86c-63638c21624f-openshift-service-ca\") pod \"maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.245227 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.245027 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5wtw\" (UniqueName: \"kubernetes.io/projected/b3e426fb-8668-4ca2-b86c-63638c21624f-kube-api-access-s5wtw\") pod \"maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.245227 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.245055 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-metrics-certs\") pod \"maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.245685 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.245654 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e426fb-8668-4ca2-b86c-63638c21624f-openshift-service-ca\") pod \"maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.247645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.247622 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-metrics-certs\") pod \"maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.247726 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.247660 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-maas-api-tls\") pod \"maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.267905 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.267875 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5wtw\" (UniqueName: \"kubernetes.io/projected/b3e426fb-8668-4ca2-b86c-63638c21624f-kube-api-access-s5wtw\") pod \"maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.302054 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.301968 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:44.350574 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.350537 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:44.648967 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:44.648937 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq"] Nov 02 07:34:44.651465 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:34:44.651437 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e426fb_8668_4ca2_b86c_63638c21624f.slice/crio-74b008a78fce7485418b0b0cc0565fc14aca2de45aef694d98309fda79850d92 WatchSource:0}: Error finding container 74b008a78fce7485418b0b0cc0565fc14aca2de45aef694d98309fda79850d92: Status 404 returned error can't find the container with id 74b008a78fce7485418b0b0cc0565fc14aca2de45aef694d98309fda79850d92 Nov 02 07:34:45.194342 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:45.194118 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:45.333025 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:45.332954 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:45.555531 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:45.555491 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" event={"ID":"b3e426fb-8668-4ca2-b86c-63638c21624f","Type":"ContainerStarted","Data":"74b008a78fce7485418b0b0cc0565fc14aca2de45aef694d98309fda79850d92"} Nov 02 07:34:46.177596 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:46.177559 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:46.315826 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:46.315794 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:47.566699 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:47.566653 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" event={"ID":"b3e426fb-8668-4ca2-b86c-63638c21624f","Type":"ContainerStarted","Data":"a7c4a288672d2d062457c67b25eeb927b85c89e00f970d239ce16107cd112b62"} Nov 02 07:34:47.567079 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:47.566761 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:47.607852 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:47.607802 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" podStartSLOduration=1.323137656 podStartE2EDuration="3.60778828s" totalImagesPullingTime="2.284650624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:47.606940976 +0000 UTC m=+587.905221813" watchObservedRunningTime="2026-11-02 07:34:47.60778828 +0000 UTC m=+587.906069104" Nov 02 07:34:48.134960 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:48.134899 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:49.952999 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:49.952965 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:51.984225 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:51.984189 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:52.116416 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:52.116378 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:52.259582 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:52.259500 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:53.546350 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:53.546323 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" Nov 02 07:34:53.546745 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:53.546389 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" Nov 02 07:34:53.576648 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:53.576622 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:34:54.264507 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:54.264477 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:56.304403 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:56.304372 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:34:58.438601 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:34:58.438538 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:00.521872 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.521836 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:00.537087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.537057 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f"] Nov 02 07:35:00.541929 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.541892 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" Nov 02 07:35:00.559373 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.559344 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f"] Nov 02 07:35:00.569200 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.569177 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:35:00.569342 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.569235 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\"" Nov 02 07:35:00.602818 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.602784 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f\" (UID: \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" Nov 02 07:35:00.602988 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.602832 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqcng\" (UniqueName: \"kubernetes.io/projected/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-kube-api-access-qqcng\") pod \"maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f\" (UID: \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" Nov 02 07:35:00.602988 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.602941 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f\" (UID: \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" Nov 02 07:35:00.703973 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.703935 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f\" (UID: \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" Nov 02 07:35:00.704231 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.704042 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f\" (UID: \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" Nov 02 07:35:00.704231 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.704084 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcng\" (UniqueName: \"kubernetes.io/projected/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-kube-api-access-qqcng\") pod \"maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f\" (UID: \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" Nov 02 07:35:00.704640 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.704619 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f\" (UID: \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" Nov 02 07:35:00.706366 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.706349 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f\" (UID: \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" Nov 02 07:35:00.727197 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.727148 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqcng\" (UniqueName: \"kubernetes.io/projected/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-kube-api-access-qqcng\") pod \"maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f\" (UID: \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" Nov 02 07:35:00.752722 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.752694 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:00.857855 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:00.857761 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" Nov 02 07:35:01.153091 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:01.153014 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f"] Nov 02 07:35:01.155286 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:35:01.155254 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7d89b3_dc0f_4ff9_9b2b_c15916e086a8.slice/crio-6e7533c482772da886210098cb90b4e519a70dd52fa7b9a971f766f4727906e7 WatchSource:0}: Error finding container 6e7533c482772da886210098cb90b4e519a70dd52fa7b9a971f766f4727906e7: Status 404 returned error can't find the container with id 6e7533c482772da886210098cb90b4e519a70dd52fa7b9a971f766f4727906e7 Nov 02 07:35:01.634105 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:01.634070 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" event={"ID":"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8","Type":"ContainerStarted","Data":"6e7533c482772da886210098cb90b4e519a70dd52fa7b9a971f766f4727906e7"} Nov 02 07:35:02.440739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:02.440708 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:04.328430 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:04.328392 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:04.650551 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:04.650513 5899 generic.go:356] "Generic (PLEG): container finished" podID="9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8" containerID="6c9e3f4e8b16f1bf71005613b763cf47560e8745ade1da1c6c7d5441ed01838e" exitCode=0 Nov 02 07:35:04.650742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:04.650606 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" event={"ID":"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8","Type":"ContainerDied","Data":"6c9e3f4e8b16f1bf71005613b763cf47560e8745ade1da1c6c7d5441ed01838e"} Nov 02 07:35:05.370032 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:05.369999 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:05.517714 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:05.517684 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:05.792447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:05.792424 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" Nov 02 07:35:05.856051 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:05.856015 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-kube-api-access-qqcng\" (UniqueName: \"kubernetes.io/projected/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-kube-api-access-qqcng\") pod \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\" (UID: \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\") " Nov 02 07:35:05.856260 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:05.856148 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-xks-ca\") pod \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\" (UID: \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\") " Nov 02 07:35:05.856260 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:05.856232 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-openshift-ca\") pod \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\" (UID: \"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8\") " Nov 02 07:35:05.856674 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:05.856648 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-openshift-ca" pod "9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8" (UID: "9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:35:05.858522 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:05.858477 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-xks-ca" pod "9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8" (UID: "9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:35:05.858636 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:05.858587 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-kube-api-access-qqcng" pod "9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8" (UID: "9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8"). InnerVolumeSpecName "kube-api-access-qqcng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:35:05.957228 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:05.957134 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqcng\" (UniqueName: \"kubernetes.io/projected/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-kube-api-access-qqcng\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:35:05.957228 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:05.957188 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:35:05.957228 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:05.957199 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:35:06.286532 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:06.286497 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:06.663869 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:06.663790 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" Nov 02 07:35:06.663869 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:06.663806 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f" event={"ID":"9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8","Type":"ContainerDied","Data":"6e7533c482772da886210098cb90b4e519a70dd52fa7b9a971f766f4727906e7"} Nov 02 07:35:06.663869 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:06.663851 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e7533c482772da886210098cb90b4e519a70dd52fa7b9a971f766f4727906e7" Nov 02 07:35:06.879533 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:06.879499 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg"] Nov 02 07:35:07.060608 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:07.060573 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:07.196972 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:07.196942 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:08.212465 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:08.212434 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:10.033616 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:10.033558 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:11.313142 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:11.313107 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:11.462534 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:11.462500 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:11.597028 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:11.596930 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:11.880031 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:11.879925 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" podUID="ee4374b0-f5ce-46d8-ac0a-6442a1bf7209" containerName="payload-processing" containerID="cri-o://f7f68a61dd38242c538a25d91c41a8623f595231ab1d6573d4f33c90a9801533" gracePeriod=25 Nov 02 07:35:11.951691 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:11.951658 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:12.094092 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.094054 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:12.126743 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.126717 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" Nov 02 07:35:12.216627 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.216549 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-plugins-config-volume\") pod \"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209\" (UID: \"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209\") " Nov 02 07:35:12.216627 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.216624 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-kube-api-access-w7qdq\" (UniqueName: \"kubernetes.io/projected/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-kube-api-access-w7qdq\") pod \"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209\" (UID: \"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209\") " Nov 02 07:35:12.216905 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.216880 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-plugins-config-volume" pod "ee4374b0-f5ce-46d8-ac0a-6442a1bf7209" (UID: "ee4374b0-f5ce-46d8-ac0a-6442a1bf7209"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:35:12.218728 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.218708 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-kube-api-access-w7qdq" pod "ee4374b0-f5ce-46d8-ac0a-6442a1bf7209" (UID: "ee4374b0-f5ce-46d8-ac0a-6442a1bf7209"). InnerVolumeSpecName "kube-api-access-w7qdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:35:12.234711 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.234685 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:12.317827 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.317795 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7qdq\" (UniqueName: \"kubernetes.io/projected/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-kube-api-access-w7qdq\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:35:12.317827 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.317824 5899 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209-plugins-config-volume\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:35:12.387729 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.387695 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:12.647290 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.647253 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:12.694815 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.694785 5899 generic.go:356] "Generic (PLEG): container finished" podID="ee4374b0-f5ce-46d8-ac0a-6442a1bf7209" containerID="f7f68a61dd38242c538a25d91c41a8623f595231ab1d6573d4f33c90a9801533" exitCode=0 Nov 02 07:35:12.694991 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.694830 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" Nov 02 07:35:12.694991 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.694860 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" event={"ID":"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209","Type":"ContainerDied","Data":"f7f68a61dd38242c538a25d91c41a8623f595231ab1d6573d4f33c90a9801533"} Nov 02 07:35:12.694991 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.694903 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg" event={"ID":"ee4374b0-f5ce-46d8-ac0a-6442a1bf7209","Type":"ContainerDied","Data":"4f5f64fe6085c4a35a3812511fe0e1b98a165bbc49a23e33cfa8b0a4f8d85aeb"} Nov 02 07:35:12.694991 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.694927 5899 scope.go:122] "RemoveContainer" containerID="f7f68a61dd38242c538a25d91c41a8623f595231ab1d6573d4f33c90a9801533" Nov 02 07:35:12.704578 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.704559 5899 scope.go:122] "RemoveContainer" containerID="f7f68a61dd38242c538a25d91c41a8623f595231ab1d6573d4f33c90a9801533" Nov 02 07:35:12.704835 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:35:12.704820 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f68a61dd38242c538a25d91c41a8623f595231ab1d6573d4f33c90a9801533\": container with ID starting with f7f68a61dd38242c538a25d91c41a8623f595231ab1d6573d4f33c90a9801533 not found: ID does not exist" containerID="f7f68a61dd38242c538a25d91c41a8623f595231ab1d6573d4f33c90a9801533" Nov 02 07:35:12.704880 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.704844 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f68a61dd38242c538a25d91c41a8623f595231ab1d6573d4f33c90a9801533"} err="failed to get container status \"f7f68a61dd38242c538a25d91c41a8623f595231ab1d6573d4f33c90a9801533\": rpc error: code = NotFound desc = could not find container \"f7f68a61dd38242c538a25d91c41a8623f595231ab1d6573d4f33c90a9801533\": container with ID starting with f7f68a61dd38242c538a25d91c41a8623f595231ab1d6573d4f33c90a9801533 not found: ID does not exist" Nov 02 07:35:12.743735 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.743701 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg"] Nov 02 07:35:12.747691 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.747662 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ff8e809-f85d4b49f-rt2lg"] Nov 02 07:35:12.784781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.784750 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:12.921971 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:12.921899 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:13.139617 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:35:13.139578 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:35:13.175381 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:13.175283 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:13.447327 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:13.447252 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:13.649827 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:13.649796 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:13.824671 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:13.824639 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4374b0-f5ce-46d8-ac0a-6442a1bf7209" path="/var/lib/kubelet/pods/ee4374b0-f5ce-46d8-ac0a-6442a1bf7209/volumes" Nov 02 07:35:13.863549 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:13.863487 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:14.205781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:14.205685 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:14.659597 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:14.659562 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:14.796314 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:14.796281 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:16.163800 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.163767 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq"] Nov 02 07:35:16.164235 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.163970 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" podUID="b3e426fb-8668-4ca2-b86c-63638c21624f" containerName="maas-api" containerID="cri-o://a7c4a288672d2d062457c67b25eeb927b85c89e00f970d239ce16107cd112b62" gracePeriod=30 Nov 02 07:35:16.330633 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.330594 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:16.422006 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.421945 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:35:16.560606 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.560574 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3e426fb-8668-4ca2-b86c-63638c21624f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e426fb-8668-4ca2-b86c-63638c21624f-openshift-service-ca\") pod \"b3e426fb-8668-4ca2-b86c-63638c21624f\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " Nov 02 07:35:16.560791 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.560679 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3e426fb-8668-4ca2-b86c-63638c21624f-kube-api-access-s5wtw\" (UniqueName: \"kubernetes.io/projected/b3e426fb-8668-4ca2-b86c-63638c21624f-kube-api-access-s5wtw\") pod \"b3e426fb-8668-4ca2-b86c-63638c21624f\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " Nov 02 07:35:16.560791 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.560707 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-maas-api-tls\") pod \"b3e426fb-8668-4ca2-b86c-63638c21624f\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " Nov 02 07:35:16.560791 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.560752 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-metrics-certs\") pod \"b3e426fb-8668-4ca2-b86c-63638c21624f\" (UID: \"b3e426fb-8668-4ca2-b86c-63638c21624f\") " Nov 02 07:35:16.560968 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.560909 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e426fb-8668-4ca2-b86c-63638c21624f-openshift-service-ca" pod "b3e426fb-8668-4ca2-b86c-63638c21624f" (UID: "b3e426fb-8668-4ca2-b86c-63638c21624f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:35:16.561116 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.561095 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e426fb-8668-4ca2-b86c-63638c21624f-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:35:16.562860 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.562834 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-maas-api-tls" pod "b3e426fb-8668-4ca2-b86c-63638c21624f" (UID: "b3e426fb-8668-4ca2-b86c-63638c21624f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:35:16.562860 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.562849 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-metrics-certs" pod "b3e426fb-8668-4ca2-b86c-63638c21624f" (UID: "b3e426fb-8668-4ca2-b86c-63638c21624f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:35:16.562979 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.562909 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e426fb-8668-4ca2-b86c-63638c21624f-kube-api-access-s5wtw" pod "b3e426fb-8668-4ca2-b86c-63638c21624f" (UID: "b3e426fb-8668-4ca2-b86c-63638c21624f"). InnerVolumeSpecName "kube-api-access-s5wtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:35:16.581241 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.581213 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7"] Nov 02 07:35:16.661975 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.661941 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:35:16.661975 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.661977 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5wtw\" (UniqueName: \"kubernetes.io/projected/b3e426fb-8668-4ca2-b86c-63638c21624f-kube-api-access-s5wtw\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:35:16.662202 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.661990 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3e426fb-8668-4ca2-b86c-63638c21624f-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:35:16.717941 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.717857 5899 generic.go:356] "Generic (PLEG): container finished" podID="b3e426fb-8668-4ca2-b86c-63638c21624f" containerID="a7c4a288672d2d062457c67b25eeb927b85c89e00f970d239ce16107cd112b62" exitCode=0 Nov 02 07:35:16.717941 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.717917 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" event={"ID":"b3e426fb-8668-4ca2-b86c-63638c21624f","Type":"ContainerDied","Data":"a7c4a288672d2d062457c67b25eeb927b85c89e00f970d239ce16107cd112b62"} Nov 02 07:35:16.717941 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.717925 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" Nov 02 07:35:16.718137 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.717966 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq" event={"ID":"b3e426fb-8668-4ca2-b86c-63638c21624f","Type":"ContainerDied","Data":"74b008a78fce7485418b0b0cc0565fc14aca2de45aef694d98309fda79850d92"} Nov 02 07:35:16.718137 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.717992 5899 scope.go:122] "RemoveContainer" containerID="a7c4a288672d2d062457c67b25eeb927b85c89e00f970d239ce16107cd112b62" Nov 02 07:35:16.731468 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.731440 5899 scope.go:122] "RemoveContainer" containerID="a7c4a288672d2d062457c67b25eeb927b85c89e00f970d239ce16107cd112b62" Nov 02 07:35:16.731967 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:35:16.731835 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c4a288672d2d062457c67b25eeb927b85c89e00f970d239ce16107cd112b62\": container with ID starting with a7c4a288672d2d062457c67b25eeb927b85c89e00f970d239ce16107cd112b62 not found: ID does not exist" containerID="a7c4a288672d2d062457c67b25eeb927b85c89e00f970d239ce16107cd112b62" Nov 02 07:35:16.731967 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.731879 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c4a288672d2d062457c67b25eeb927b85c89e00f970d239ce16107cd112b62"} err="failed to get container status \"a7c4a288672d2d062457c67b25eeb927b85c89e00f970d239ce16107cd112b62\": rpc error: code = NotFound desc = could not find container \"a7c4a288672d2d062457c67b25eeb927b85c89e00f970d239ce16107cd112b62\": container with ID starting with a7c4a288672d2d062457c67b25eeb927b85c89e00f970d239ce16107cd112b62 not found: ID does not exist" Nov 02 07:35:16.763934 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.763902 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:16.770907 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.770877 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq"] Nov 02 07:35:16.776550 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.776526 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-935d4e21-7d64d89649-5k4lq"] Nov 02 07:35:16.909450 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.909418 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:16.986253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.986221 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f"] Nov 02 07:35:16.989846 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:16.989820 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935d4e21-cbl6f"] Nov 02 07:35:17.054681 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:17.054636 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:17.211345 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:17.211312 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:17.530690 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:17.530650 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:17.818606 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:17.818529 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8" path="/var/lib/kubelet/pods/9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8/volumes" Nov 02 07:35:17.818947 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:17.818933 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3e426fb-8668-4ca2-b86c-63638c21624f" path="/var/lib/kubelet/pods/b3e426fb-8668-4ca2-b86c-63638c21624f/volumes" Nov 02 07:35:17.822018 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:17.821988 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:17.982567 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:17.982528 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:18.142477 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:18.142392 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:18.295241 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:18.295210 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:18.450321 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:18.450233 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:18.598715 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:18.598685 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:19.538226 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:19.538194 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:19.708994 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:19.708960 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:19.865229 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:19.865114 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:20.040351 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:20.040305 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:20.547838 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:20.547805 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:20.981667 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:20.981633 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:21.134902 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.134865 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:21.579904 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.579865 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4"] Nov 02 07:35:21.580662 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.580638 5899 cpu_manager.go:414] "containerMap: removing container" podUID="9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8" containerName="revoke-keys" Nov 02 07:35:21.580736 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.580664 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8" containerName="revoke-keys" Nov 02 07:35:21.580736 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.580716 5899 cpu_manager.go:414] "containerMap: removing container" podUID="b3e426fb-8668-4ca2-b86c-63638c21624f" containerName="maas-api" Nov 02 07:35:21.580736 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.580724 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e426fb-8668-4ca2-b86c-63638c21624f" containerName="maas-api" Nov 02 07:35:21.580895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.580744 5899 cpu_manager.go:414] "containerMap: removing container" podUID="ee4374b0-f5ce-46d8-ac0a-6442a1bf7209" containerName="payload-processing" Nov 02 07:35:21.580895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.580753 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4374b0-f5ce-46d8-ac0a-6442a1bf7209" containerName="payload-processing" Nov 02 07:35:21.580895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.580877 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7d89b3-dc0f-4ff9-9b2b-c15916e086a8" containerName="revoke-keys" Nov 02 07:35:21.581052 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.580897 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e426fb-8668-4ca2-b86c-63638c21624f" containerName="maas-api" Nov 02 07:35:21.581052 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.580907 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4374b0-f5ce-46d8-ac0a-6442a1bf7209" containerName="payload-processing" Nov 02 07:35:21.582132 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.582078 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" podUID="b32f56ab-c29b-4490-a610-39d5b95253e9" containerName="payload-processing" containerID="cri-o://3a9d1bf9797d00a63ae7cd68beb0a141e2374ea82b87df4e7e27a6f5f204ddf5" gracePeriod=25 Nov 02 07:35:21.587452 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.587429 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:21.604393 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.604372 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4tgq8\"" Nov 02 07:35:21.605700 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.605679 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:21.616062 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.616038 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 07:35:21.616228 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.616080 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-354660-serving-cert\"" Nov 02 07:35:21.617830 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.617804 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4"] Nov 02 07:35:21.716444 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.716409 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0284bc4-0507-4252-8aee-2ef6448316e8-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:21.716613 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.716462 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-metrics-certs\") pod \"maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:21.716613 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.716485 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45s4q\" (UniqueName: \"kubernetes.io/projected/b0284bc4-0507-4252-8aee-2ef6448316e8-kube-api-access-45s4q\") pod \"maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:21.716695 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.716647 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:21.747222 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.747191 5899 generic.go:356] "Generic (PLEG): container finished" podID="b32f56ab-c29b-4490-a610-39d5b95253e9" containerID="3a9d1bf9797d00a63ae7cd68beb0a141e2374ea82b87df4e7e27a6f5f204ddf5" exitCode=0 Nov 02 07:35:21.747408 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.747259 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" event={"ID":"b32f56ab-c29b-4490-a610-39d5b95253e9","Type":"ContainerDied","Data":"3a9d1bf9797d00a63ae7cd68beb0a141e2374ea82b87df4e7e27a6f5f204ddf5"} Nov 02 07:35:21.764857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.764825 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:21.817698 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.817666 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0284bc4-0507-4252-8aee-2ef6448316e8-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:21.818421 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.817733 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-metrics-certs\") pod \"maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:21.818421 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.817770 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45s4q\" (UniqueName: \"kubernetes.io/projected/b0284bc4-0507-4252-8aee-2ef6448316e8-kube-api-access-45s4q\") pod \"maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:21.818421 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.817856 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:21.819074 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.819048 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0284bc4-0507-4252-8aee-2ef6448316e8-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:21.820967 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.820943 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:21.821066 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.821007 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-metrics-certs\") pod \"maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:21.843760 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.843610 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45s4q\" (UniqueName: \"kubernetes.io/projected/b0284bc4-0507-4252-8aee-2ef6448316e8-kube-api-access-45s4q\") pod \"maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:21.852092 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.852056 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" Nov 02 07:35:21.899849 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.899817 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:21.918894 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.918857 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32f56ab-c29b-4490-a610-39d5b95253e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32f56ab-c29b-4490-a610-39d5b95253e9-plugins-config-volume\") pod \"b32f56ab-c29b-4490-a610-39d5b95253e9\" (UID: \"b32f56ab-c29b-4490-a610-39d5b95253e9\") " Nov 02 07:35:21.919078 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.918952 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32f56ab-c29b-4490-a610-39d5b95253e9-kube-api-access-wmxlt\" (UniqueName: \"kubernetes.io/projected/b32f56ab-c29b-4490-a610-39d5b95253e9-kube-api-access-wmxlt\") pod \"b32f56ab-c29b-4490-a610-39d5b95253e9\" (UID: \"b32f56ab-c29b-4490-a610-39d5b95253e9\") " Nov 02 07:35:21.919284 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.919259 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32f56ab-c29b-4490-a610-39d5b95253e9-plugins-config-volume" pod "b32f56ab-c29b-4490-a610-39d5b95253e9" (UID: "b32f56ab-c29b-4490-a610-39d5b95253e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:35:21.921084 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.921061 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32f56ab-c29b-4490-a610-39d5b95253e9-kube-api-access-wmxlt" pod "b32f56ab-c29b-4490-a610-39d5b95253e9" (UID: "b32f56ab-c29b-4490-a610-39d5b95253e9"). InnerVolumeSpecName "kube-api-access-wmxlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:35:21.937781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:21.937752 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:22.020333 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.020294 5899 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32f56ab-c29b-4490-a610-39d5b95253e9-plugins-config-volume\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:35:22.020333 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.020335 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmxlt\" (UniqueName: \"kubernetes.io/projected/b32f56ab-c29b-4490-a610-39d5b95253e9-kube-api-access-wmxlt\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:35:22.070988 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.070942 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:22.211267 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.211190 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:22.240060 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.240033 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4"] Nov 02 07:35:22.754155 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.754113 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" event={"ID":"b0284bc4-0507-4252-8aee-2ef6448316e8","Type":"ContainerStarted","Data":"9d5a46a8014c0e9f7ed63ad28ce1a9d571c6795634cb787771280b01736337c1"} Nov 02 07:35:22.754613 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.754203 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" event={"ID":"b0284bc4-0507-4252-8aee-2ef6448316e8","Type":"ContainerStarted","Data":"a45732457bda392f288d47de2bb543691684d8956fd9cf593faa3b080efa29b0"} Nov 02 07:35:22.754613 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.754240 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:22.755444 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.755424 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" event={"ID":"b32f56ab-c29b-4490-a610-39d5b95253e9","Type":"ContainerDied","Data":"a51fb1a377a0f4e61d9cb58c61eddfad566a841fc1c357c35a32e44873b8ae18"} Nov 02 07:35:22.755520 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.755456 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7" Nov 02 07:35:22.755520 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.755472 5899 scope.go:122] "RemoveContainer" containerID="3a9d1bf9797d00a63ae7cd68beb0a141e2374ea82b87df4e7e27a6f5f204ddf5" Nov 02 07:35:22.795432 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.795388 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" podStartSLOduration=1.480387724 podStartE2EDuration="1.795373754s" totalImagesPullingTime="314.98603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:22.794495859 +0000 UTC m=+623.092776703" watchObservedRunningTime="2026-11-02 07:35:22.795373754 +0000 UTC m=+623.093654579" Nov 02 07:35:22.831748 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.831653 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7"] Nov 02 07:35:22.833886 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:22.833861 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-be1fb7c9-868cbddff7-qq8s7"] Nov 02 07:35:23.501487 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:23.501448 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:23.818335 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:23.818257 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32f56ab-c29b-4490-a610-39d5b95253e9" path="/var/lib/kubelet/pods/b32f56ab-c29b-4490-a610-39d5b95253e9/volumes" Nov 02 07:35:24.098790 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.098688 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:24.150504 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.150463 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk"] Nov 02 07:35:24.151154 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.151123 5899 cpu_manager.go:414] "containerMap: removing container" podUID="b32f56ab-c29b-4490-a610-39d5b95253e9" containerName="payload-processing" Nov 02 07:35:24.151154 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.151150 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32f56ab-c29b-4490-a610-39d5b95253e9" containerName="payload-processing" Nov 02 07:35:24.151333 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.151278 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32f56ab-c29b-4490-a610-39d5b95253e9" containerName="payload-processing" Nov 02 07:35:24.155893 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.155875 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.184795 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.183357 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-0fbc17-serving-cert\"" Nov 02 07:35:24.185137 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.185113 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk"] Nov 02 07:35:24.245303 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.245267 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/596b979e-253a-4286-88ca-e0f737d3be85-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.245476 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.245330 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-metrics-certs\") pod \"maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.245476 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.245462 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.245604 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.245589 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt6xj\" (UniqueName: \"kubernetes.io/projected/596b979e-253a-4286-88ca-e0f737d3be85-kube-api-access-kt6xj\") pod \"maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.246037 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.246014 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:24.346563 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.346524 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6xj\" (UniqueName: \"kubernetes.io/projected/596b979e-253a-4286-88ca-e0f737d3be85-kube-api-access-kt6xj\") pod \"maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.346727 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.346592 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/596b979e-253a-4286-88ca-e0f737d3be85-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.346727 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.346651 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-metrics-certs\") pod \"maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.346727 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.346706 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.347567 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.347356 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/596b979e-253a-4286-88ca-e0f737d3be85-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.349483 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.349425 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-metrics-certs\") pod \"maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.349577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.349508 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.369077 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.369052 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6xj\" (UniqueName: \"kubernetes.io/projected/596b979e-253a-4286-88ca-e0f737d3be85-kube-api-access-kt6xj\") pod \"maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.385464 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.385428 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:24.469896 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.469858 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:24.517449 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.517417 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:24.661281 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.661196 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:24.769917 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.769893 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk"] Nov 02 07:35:24.772501 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:35:24.772474 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596b979e_253a_4286_88ca_e0f737d3be85.slice/crio-f7e118d9e9d01bf5f46db77f3985a5cd602c597f29f50955ded52ddcc268f9dd WatchSource:0}: Error finding container f7e118d9e9d01bf5f46db77f3985a5cd602c597f29f50955ded52ddcc268f9dd: Status 404 returned error can't find the container with id f7e118d9e9d01bf5f46db77f3985a5cd602c597f29f50955ded52ddcc268f9dd Nov 02 07:35:24.800325 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.800286 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:24.981415 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:24.981382 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:25.618446 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:25.618415 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:25.769826 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:25.769793 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:25.774712 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:25.774681 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" event={"ID":"596b979e-253a-4286-88ca-e0f737d3be85","Type":"ContainerStarted","Data":"415e46d5b4299f667ddcf931e46bf2f1ceae7e8b6ee6e42d1b22449cdc6bd31b"} Nov 02 07:35:25.774860 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:25.774740 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" event={"ID":"596b979e-253a-4286-88ca-e0f737d3be85","Type":"ContainerStarted","Data":"f7e118d9e9d01bf5f46db77f3985a5cd602c597f29f50955ded52ddcc268f9dd"} Nov 02 07:35:25.774981 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:25.774960 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:25.819865 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:25.819815 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" podStartSLOduration=1.503034243 podStartE2EDuration="1.819800621s" totalImagesPullingTime="316.766378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:25.818277311 +0000 UTC m=+626.116558137" watchObservedRunningTime="2026-11-02 07:35:25.819800621 +0000 UTC m=+626.118081445" Nov 02 07:35:26.237413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:26.237380 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:26.377183 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:26.376911 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:28.091552 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:28.091521 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:28.236671 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:28.236641 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:28.382271 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:28.382195 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:28.521895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:28.521859 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:28.766456 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:28.766425 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 07:35:29.754435 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:29.754400 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:29.898801 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:29.898767 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:30.299444 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:30.299385 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:31.460262 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:31.460227 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:31.623208 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:31.623156 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:31.786839 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:31.786809 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 07:35:32.016719 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.016684 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr"] Nov 02 07:35:32.020557 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.020525 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.034845 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.034818 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-9db802-serving-cert\"" Nov 02 07:35:32.041047 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.040981 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr"] Nov 02 07:35:32.124646 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.124617 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-metrics-certs\") pod \"maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.124814 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.124656 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.124814 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.124767 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c41bdc-daa2-4024-8042-cb6810f56988-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.124916 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.124833 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftss\" (UniqueName: \"kubernetes.io/projected/c9c41bdc-daa2-4024-8042-cb6810f56988-kube-api-access-bftss\") pod \"maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.148711 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.148681 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:32.226330 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.226293 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-metrics-certs\") pod \"maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.226330 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.226335 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.226577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.226381 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c41bdc-daa2-4024-8042-cb6810f56988-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.226577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.226433 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftss\" (UniqueName: \"kubernetes.io/projected/c9c41bdc-daa2-4024-8042-cb6810f56988-kube-api-access-bftss\") pod \"maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.227125 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.227104 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c41bdc-daa2-4024-8042-cb6810f56988-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.228760 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.228738 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-metrics-certs\") pod \"maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.228841 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.228795 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.255613 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.255584 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftss\" (UniqueName: \"kubernetes.io/projected/c9c41bdc-daa2-4024-8042-cb6810f56988-kube-api-access-bftss\") pod \"maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.291269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.291174 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:32.333787 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.333747 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:32.637877 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.637849 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr"] Nov 02 07:35:32.640746 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:35:32.640723 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c41bdc_daa2_4024_8042_cb6810f56988.slice/crio-b1a810d22d9d046d962869ee32a17f03aebfc918271cbd3bac3de927aa90f765 WatchSource:0}: Error finding container b1a810d22d9d046d962869ee32a17f03aebfc918271cbd3bac3de927aa90f765: Status 404 returned error can't find the container with id b1a810d22d9d046d962869ee32a17f03aebfc918271cbd3bac3de927aa90f765 Nov 02 07:35:32.642056 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.642038 5899 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 07:35:32.818817 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:32.818776 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" event={"ID":"c9c41bdc-daa2-4024-8042-cb6810f56988","Type":"ContainerStarted","Data":"b1a810d22d9d046d962869ee32a17f03aebfc918271cbd3bac3de927aa90f765"} Nov 02 07:35:33.824450 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:33.824408 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" event={"ID":"c9c41bdc-daa2-4024-8042-cb6810f56988","Type":"ContainerStarted","Data":"5386296fc8130e1cd4d30b5d05778b991c34bca764429ee73e6196363fa1beb0"} Nov 02 07:35:33.824918 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:33.824556 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:33.865292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:33.865238 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" podStartSLOduration=2.444577878 podStartE2EDuration="2.865226107s" totalImagesPullingTime="420.648229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:33.864974232 +0000 UTC m=+634.163255057" watchObservedRunningTime="2026-11-02 07:35:33.865226107 +0000 UTC m=+634.163506932" Nov 02 07:35:34.349994 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:34.349963 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:36.223919 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:36.223880 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:37.761885 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:37.761849 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:38.337649 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:38.337605 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:38.484336 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:38.484298 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:38.630109 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:38.630032 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:39.340989 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:39.340951 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:39.835034 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:39.835001 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 07:35:39.970415 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:39.970377 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:40.112262 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.112138 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:40.451754 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.451678 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr"] Nov 02 07:35:40.455738 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.455718 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:40.485934 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.485850 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr"] Nov 02 07:35:40.489291 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.489265 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-7e7a83-serving-cert\"" Nov 02 07:35:40.515356 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.514878 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7b8863-0ace-444b-ab7d-2220f7f3137a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:40.515356 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.515029 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrnz7\" (UniqueName: \"kubernetes.io/projected/da7b8863-0ace-444b-ab7d-2220f7f3137a-kube-api-access-hrnz7\") pod \"maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:40.515356 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.515071 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-metrics-certs\") pod \"maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:40.515356 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.515216 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:40.600255 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.600217 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:40.616472 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.616443 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7b8863-0ace-444b-ab7d-2220f7f3137a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:40.616639 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.616494 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrnz7\" (UniqueName: \"kubernetes.io/projected/da7b8863-0ace-444b-ab7d-2220f7f3137a-kube-api-access-hrnz7\") pod \"maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:40.616639 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.616526 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-metrics-certs\") pod \"maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:40.616639 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.616600 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:40.617318 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.617289 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7b8863-0ace-444b-ab7d-2220f7f3137a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:40.619213 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.619188 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:40.619298 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.619212 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-metrics-certs\") pod \"maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:40.644225 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.644198 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrnz7\" (UniqueName: \"kubernetes.io/projected/da7b8863-0ace-444b-ab7d-2220f7f3137a-kube-api-access-hrnz7\") pod \"maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:40.734382 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.734349 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:40.769660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:40.769629 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:41.046831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:41.046743 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:41.078583 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:41.078553 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr"] Nov 02 07:35:41.081188 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:35:41.081141 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7b8863_0ace_444b_ab7d_2220f7f3137a.slice/crio-0f70d8771ea4b97a21a1deadad28f7c5fdccd12a1f3b04407777a8864a368a82 WatchSource:0}: Error finding container 0f70d8771ea4b97a21a1deadad28f7c5fdccd12a1f3b04407777a8864a368a82: Status 404 returned error can't find the container with id 0f70d8771ea4b97a21a1deadad28f7c5fdccd12a1f3b04407777a8864a368a82 Nov 02 07:35:41.601532 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:41.601504 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:41.866304 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:41.866217 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" event={"ID":"da7b8863-0ace-444b-ab7d-2220f7f3137a","Type":"ContainerStarted","Data":"ff8025d780f2fc04f036e367547bd4c2c9136f5bea0c3e31975c50fb9b3f8b41"} Nov 02 07:35:41.866304 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:41.866272 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" event={"ID":"da7b8863-0ace-444b-ab7d-2220f7f3137a","Type":"ContainerStarted","Data":"0f70d8771ea4b97a21a1deadad28f7c5fdccd12a1f3b04407777a8864a368a82"} Nov 02 07:35:41.866481 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:41.866350 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:41.912547 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:41.912484 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" podStartSLOduration=1.469158172 podStartE2EDuration="1.912467735s" totalImagesPullingTime="443.309563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:41.910876503 +0000 UTC m=+642.209157328" watchObservedRunningTime="2026-11-02 07:35:41.912467735 +0000 UTC m=+642.210748561" Nov 02 07:35:42.176109 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:42.176019 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:42.318271 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:42.318226 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:42.467501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:42.467419 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:42.802648 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:42.802618 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:43.141201 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:35:43.141085 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:35:44.553213 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:44.553181 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:46.506941 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:46.506900 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:47.877549 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:47.877520 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 07:35:48.663327 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:48.663296 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:48.895433 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:48.895395 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:49.665294 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:49.665252 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:50.228089 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:50.228055 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:50.367292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:50.367248 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:50.510486 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:50.510382 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:50.661133 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:50.661091 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:50.805245 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:50.805145 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:52.744624 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:52.744586 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:54.796367 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:54.796326 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:56.713712 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:56.713676 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:58.531710 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.531671 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:58.729109 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.729075 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt"] Nov 02 07:35:58.734009 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.733986 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.763065 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.762910 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9db802\"/\"default-dockercfg-pnzx7\"" Nov 02 07:35:58.763065 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.762910 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9db802\"/\"openshift-service-ca.crt\"" Nov 02 07:35:58.763065 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.762919 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9db802\"/\"kube-root-ca.crt\"" Nov 02 07:35:58.763065 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.763028 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9db802\"/\"simulator-w4-9db802-kserve-self-signed-certs\"" Nov 02 07:35:58.765608 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.765586 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt"] Nov 02 07:35:58.796398 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.796322 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-tmp-dir\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.796398 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.796391 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-dshm\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.796582 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.796473 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljlxp\" (UniqueName: \"kubernetes.io/projected/9be63900-74ab-48d6-aee0-6071d5d2a305-kube-api-access-ljlxp\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.796582 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.796538 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-model-cache\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.796659 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.796631 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-home\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.796659 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.796653 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be63900-74ab-48d6-aee0-6071d5d2a305-tls-certs\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.815046 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.815012 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:58.897732 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.897694 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-tmp-dir\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.897943 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.897773 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-dshm\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.897943 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.897809 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljlxp\" (UniqueName: \"kubernetes.io/projected/9be63900-74ab-48d6-aee0-6071d5d2a305-kube-api-access-ljlxp\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.897943 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.897870 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-model-cache\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.897943 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.897896 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-home\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.897943 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.897920 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be63900-74ab-48d6-aee0-6071d5d2a305-tls-certs\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.898283 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.898217 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-tmp-dir\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.898486 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.898460 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-model-cache\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.898610 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.898477 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-home\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.900706 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.900685 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-dshm\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.901072 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.901048 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be63900-74ab-48d6-aee0-6071d5d2a305-tls-certs\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:58.921030 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:58.921000 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljlxp\" (UniqueName: \"kubernetes.io/projected/9be63900-74ab-48d6-aee0-6071d5d2a305-kube-api-access-ljlxp\") pod \"simulator-w4-9db802-kserve-65676b8468-vrmnt\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:59.047951 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:59.047857 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:35:59.344910 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:59.344883 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt"] Nov 02 07:35:59.348292 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:35:59.348256 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9be63900_74ab_48d6_aee0_6071d5d2a305.slice/crio-0aff9e60bf8fbfb922762c94216e8c2061e4dd5b4baaf18fec8aa9112662723c WatchSource:0}: Error finding container 0aff9e60bf8fbfb922762c94216e8c2061e4dd5b4baaf18fec8aa9112662723c: Status 404 returned error can't find the container with id 0aff9e60bf8fbfb922762c94216e8c2061e4dd5b4baaf18fec8aa9112662723c Nov 02 07:35:59.911812 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:59.911782 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:35:59.963292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:35:59.963249 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" event={"ID":"9be63900-74ab-48d6-aee0-6071d5d2a305","Type":"ContainerStarted","Data":"0aff9e60bf8fbfb922762c94216e8c2061e4dd5b4baaf18fec8aa9112662723c"} Nov 02 07:36:00.429147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:00.429090 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:00.566034 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:00.565998 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:00.701051 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:00.700959 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:00.899648 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:00.899615 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:00.969228 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:00.969125 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" event={"ID":"9be63900-74ab-48d6-aee0-6071d5d2a305","Type":"ContainerStarted","Data":"0e85ee29cc7daadec8137eac868fd20cd6371c848f53f4a8832c8e755541cc34"} Nov 02 07:36:01.013773 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:01.013717 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" podStartSLOduration=1.790702584 podStartE2EDuration="3.013701123s" totalImagesPullingTime="1.222998539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:36:01.013324966 +0000 UTC m=+661.311605793" watchObservedRunningTime="2026-11-02 07:36:01.013701123 +0000 UTC m=+661.311981952" Nov 02 07:36:01.048275 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:01.048241 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:36:02.457659 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:02.457620 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:04.310050 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:04.310016 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:04.787727 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:04.787675 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:06.110510 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:06.110477 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:07.815284 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:07.815247 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:09.048312 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:09.048278 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:36:09.061465 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:09.061439 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:36:09.607352 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:09.607316 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:09.794589 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:09.794519 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:10.024246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:10.024220 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:36:11.415334 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:11.415287 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:13.102229 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:13.102192 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:13.144018 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:36:13.143982 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:36:14.923339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:14.923303 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:16.170802 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:16.170767 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:16.709211 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:16.709180 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:18.416559 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:18.416524 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:20.110679 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:20.110641 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:21.804422 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:21.804383 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:22.028118 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.028080 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k"] Nov 02 07:36:22.034929 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.034900 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.055402 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.055329 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k"] Nov 02 07:36:22.061403 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.061369 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-337764\"/\"kube-root-ca.crt\"" Nov 02 07:36:22.061403 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.061368 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-337764\"/\"openshift-service-ca.crt\"" Nov 02 07:36:22.061641 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.061471 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-337764\"/\"premium-w2-337764-kserve-self-signed-certs\"" Nov 02 07:36:22.061722 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.061690 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-337764\"/\"default-dockercfg-5lt4c\"" Nov 02 07:36:22.119179 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.119135 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:22.123263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.123231 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-home\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.123421 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.123353 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-tmp-dir\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.123421 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.123370 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-dshm\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.123544 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.123445 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-model-cache\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.123544 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.123483 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pgn2\" (UniqueName: \"kubernetes.io/projected/de594282-67a5-4dc7-8170-53abd0c34041-kube-api-access-2pgn2\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.123652 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.123609 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de594282-67a5-4dc7-8170-53abd0c34041-tls-certs\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.224277 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.224232 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-tmp-dir\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.224589 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.224566 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-dshm\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.224715 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.224610 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-model-cache\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.224715 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.224637 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pgn2\" (UniqueName: \"kubernetes.io/projected/de594282-67a5-4dc7-8170-53abd0c34041-kube-api-access-2pgn2\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.224715 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.224693 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de594282-67a5-4dc7-8170-53abd0c34041-tls-certs\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.224881 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.224743 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-home\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.224942 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.224875 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-tmp-dir\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.225386 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.225143 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-home\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.225386 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.225344 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-model-cache\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.235341 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.227809 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-dshm\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.235341 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.228034 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de594282-67a5-4dc7-8170-53abd0c34041-tls-certs\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.247345 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.247316 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pgn2\" (UniqueName: \"kubernetes.io/projected/de594282-67a5-4dc7-8170-53abd0c34041-kube-api-access-2pgn2\") pod \"premium-w2-337764-kserve-bc9d49c4-p5m7k\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.356391 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.356296 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:22.652923 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:22.652889 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k"] Nov 02 07:36:22.655777 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:36:22.655747 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde594282_67a5_4dc7_8170_53abd0c34041.slice/crio-573e2a67828809337f2a08972f5a83eadaf47ca9a3e4ec748ffe6cf1964e7dca WatchSource:0}: Error finding container 573e2a67828809337f2a08972f5a83eadaf47ca9a3e4ec748ffe6cf1964e7dca: Status 404 returned error can't find the container with id 573e2a67828809337f2a08972f5a83eadaf47ca9a3e4ec748ffe6cf1964e7dca Nov 02 07:36:23.082251 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:23.082211 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" event={"ID":"de594282-67a5-4dc7-8170-53abd0c34041","Type":"ContainerStarted","Data":"ecc91f62c8cbdc066e473317d4ade274246ef7a18010283402b1b37848c771d8"} Nov 02 07:36:23.082630 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:23.082266 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" event={"ID":"de594282-67a5-4dc7-8170-53abd0c34041","Type":"ContainerStarted","Data":"573e2a67828809337f2a08972f5a83eadaf47ca9a3e4ec748ffe6cf1964e7dca"} Nov 02 07:36:23.124220 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:23.124149 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podStartSLOduration=1.124135139 podStartE2EDuration="1.124135139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:36:23.123277365 +0000 UTC m=+683.421558192" watchObservedRunningTime="2026-11-02 07:36:23.124135139 +0000 UTC m=+683.422415964" Nov 02 07:36:23.356525 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:23.356441 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:23.694261 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:23.694125 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:25.405687 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:25.405642 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:27.211709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:27.211667 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:28.948453 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:28.948421 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:30.173201 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.171609 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc"] Nov 02 07:36:30.176082 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.176058 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.202532 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.202497 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-model-cache\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.202704 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.202627 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tmp-dir\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.202774 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.202706 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skqsz\" (UniqueName: \"kubernetes.io/projected/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-kube-api-access-skqsz\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.202848 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.202830 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tls-certs\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.202929 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.202913 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-home\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.202985 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.202968 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-dshm\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.205549 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.205523 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9db802\"/\"premium-w4-9db802-kserve-self-signed-certs\"" Nov 02 07:36:30.207382 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.207359 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc"] Nov 02 07:36:30.252581 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.252544 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:30.304431 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.304392 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tmp-dir\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.304620 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.304440 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skqsz\" (UniqueName: \"kubernetes.io/projected/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-kube-api-access-skqsz\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.304620 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.304548 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tls-certs\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.304620 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.304612 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-home\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.304781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.304639 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-dshm\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.304781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.304670 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-model-cache\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.304947 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.304918 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tmp-dir\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.305019 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.304982 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-model-cache\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.305175 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.305140 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-home\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.307033 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.307003 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-dshm\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.307033 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.307022 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tls-certs\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.327084 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.327054 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skqsz\" (UniqueName: \"kubernetes.io/projected/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-kube-api-access-skqsz\") pod \"premium-w4-9db802-kserve-5496bf967c-zdmdc\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.491228 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.491200 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:30.689299 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.689267 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:30.791976 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:30.791942 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc"] Nov 02 07:36:30.794588 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:36:30.794560 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6bd9f32_cb7a_4a67_b7bb_cdf1d89ea418.slice/crio-81270840f7295d569f35be9ece694311bc11ab575c30d6d4f1af163ad6f24ed3 WatchSource:0}: Error finding container 81270840f7295d569f35be9ece694311bc11ab575c30d6d4f1af163ad6f24ed3: Status 404 returned error can't find the container with id 81270840f7295d569f35be9ece694311bc11ab575c30d6d4f1af163ad6f24ed3 Nov 02 07:36:31.124116 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:31.124030 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" event={"ID":"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418","Type":"ContainerStarted","Data":"6eb612ad0bec54e0d4043861ff391bbcfe53fb4eca204c976894325dd6728e05"} Nov 02 07:36:31.124116 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:31.124072 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" event={"ID":"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418","Type":"ContainerStarted","Data":"81270840f7295d569f35be9ece694311bc11ab575c30d6d4f1af163ad6f24ed3"} Nov 02 07:36:31.164637 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:31.164586 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" podStartSLOduration=1.164574543 podStartE2EDuration="1.164574543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:36:31.163913439 +0000 UTC m=+691.462194264" watchObservedRunningTime="2026-11-02 07:36:31.164574543 +0000 UTC m=+691.462855368" Nov 02 07:36:31.491315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:31.491282 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:32.356693 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:32.356655 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:32.369271 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:32.369237 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:32.641931 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:32.641854 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:33.149246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:33.149214 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:36:34.604691 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:34.604657 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:36.214641 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:36.214589 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:37.905189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:37.905133 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:39.620282 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:39.620251 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:39.948001 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:39.947906 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:40.491780 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:40.491745 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:40.504677 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:40.504647 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:41.192867 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:41.192822 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 07:36:41.508293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:41.508260 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:42.523065 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:42.523031 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:43.141568 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:36:43.141526 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:36:43.203796 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:43.203761 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:45.002137 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:45.002101 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:46.817214 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:46.817180 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:48.594405 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:48.594368 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:50.584364 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:50.584329 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:52.246984 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:52.246948 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:53.920780 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:53.920716 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:55.700059 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:55.700017 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:57.402579 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:57.402546 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:36:59.187410 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:36:59.187368 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:01.001455 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:01.001423 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:01.144912 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:01.144833 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:03.027262 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:03.027227 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:04.832669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:04.832637 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:06.618351 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:06.618314 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:08.409833 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:08.409800 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:10.274345 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:10.274310 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:12.190498 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:12.190452 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:13.141660 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:37:13.141620 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:37:13.903319 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:13.903278 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:15.608422 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:15.608388 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:17.414481 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:17.414447 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:19.162756 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:19.162716 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:20.890473 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:20.890438 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:22.703172 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:22.703127 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:24.614785 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:24.614747 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:26.414946 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:26.414906 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:28.112900 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:28.112861 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:29.904278 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:29.904237 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:31.601986 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:31.601950 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:32.597807 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:32.597754 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:32.738871 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:32.738839 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:33.421921 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:33.421884 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:34.032260 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:34.032221 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:35.300325 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:35.300268 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:37.121675 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:37.121629 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:39.010009 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:39.009960 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:40.810855 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:40.810818 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:42.694219 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:42.694153 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:43.143813 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:37:43.143780 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:37:44.504094 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:44.504058 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:46.203516 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:46.203470 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:48.025337 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.025302 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:48.143918 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.143888 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx"] Nov 02 07:37:48.148452 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.148427 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.171382 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.171352 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx"] Nov 02 07:37:48.287790 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.287696 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-495k7\" (UniqueName: \"kubernetes.io/projected/e43d749b-223f-4a0a-997a-a0637ddf2d00-kube-api-access-495k7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.287790 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.287750 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.288009 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.287857 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.288009 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.287901 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.288082 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.288000 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43d749b-223f-4a0a-997a-a0637ddf2d00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.288082 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.288040 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.288082 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.288061 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.288273 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.288134 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43d749b-223f-4a0a-997a-a0637ddf2d00-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.288273 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.288196 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43d749b-223f-4a0a-997a-a0637ddf2d00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.389089 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.389030 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495k7\" (UniqueName: \"kubernetes.io/projected/e43d749b-223f-4a0a-997a-a0637ddf2d00-kube-api-access-495k7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.389305 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.389101 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.389305 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.389154 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.389305 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.389229 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.389305 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.389271 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43d749b-223f-4a0a-997a-a0637ddf2d00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.389530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.389310 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.389530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.389339 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.389530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.389407 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43d749b-223f-4a0a-997a-a0637ddf2d00-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.389530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.389433 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43d749b-223f-4a0a-997a-a0637ddf2d00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.389530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.389509 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.389788 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.389552 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.389788 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.389655 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.389963 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.389795 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.390192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.390147 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43d749b-223f-4a0a-997a-a0637ddf2d00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.391976 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.391952 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43d749b-223f-4a0a-997a-a0637ddf2d00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.392066 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.392045 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43d749b-223f-4a0a-997a-a0637ddf2d00-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.412924 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.412886 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43d749b-223f-4a0a-997a-a0637ddf2d00-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.413079 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.413064 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-495k7\" (UniqueName: \"kubernetes.io/projected/e43d749b-223f-4a0a-997a-a0637ddf2d00-kube-api-access-495k7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-7xqxx\" (UID: \"e43d749b-223f-4a0a-997a-a0637ddf2d00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.462013 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.461979 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:48.767816 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.767782 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6"] Nov 02 07:37:48.773305 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.773280 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.780933 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.780909 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx"] Nov 02 07:37:48.783883 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:37:48.783856 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43d749b_223f_4a0a_997a_a0637ddf2d00.slice/crio-a57d154d3261e326487e6ca073e2aface619e6e971f729f6eefcb39a55e58283 WatchSource:0}: Error finding container a57d154d3261e326487e6ca073e2aface619e6e971f729f6eefcb39a55e58283: Status 404 returned error can't find the container with id a57d154d3261e326487e6ca073e2aface619e6e971f729f6eefcb39a55e58283 Nov 02 07:37:48.786211 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.786184 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:37:48.786285 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.786243 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:37:48.786285 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.786270 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:37:48.790350 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.790331 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-337764-openshift-default-dockercfg-45fm2\"" Nov 02 07:37:48.796073 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.796049 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6"] Nov 02 07:37:48.894472 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.894436 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.894638 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.894484 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.894710 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.894642 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.894770 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.894738 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.894856 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.894840 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.894919 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.894873 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.894969 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.894915 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhjs\" (UniqueName: \"kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-kube-api-access-glhjs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.895012 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.894993 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.895063 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.895046 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.995857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.995817 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.995857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.995868 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.996111 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.995910 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhjs\" (UniqueName: \"kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-kube-api-access-glhjs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.996111 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.995940 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.996317 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.996289 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.996449 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.996409 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.996530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.996412 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.996596 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.996553 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.996596 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.996583 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.996702 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.996659 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.997060 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.996864 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.997060 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.996971 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.997060 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.996983 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.997060 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.997029 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.999276 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.999249 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:48.999389 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:48.999369 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:49.019715 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.019654 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:49.019819 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.019770 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhjs\" (UniqueName: \"kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-kube-api-access-glhjs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:49.090622 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.090588 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:49.408471 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.408428 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6"] Nov 02 07:37:49.411358 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:37:49.411325 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0427a3_509d_4e27_8cf2_e5782fd4ff0e.slice/crio-4e763662b888a6a2fd4e9c24bafa4d764b0ef0544a0366be6cfb845d3141b411 WatchSource:0}: Error finding container 4e763662b888a6a2fd4e9c24bafa4d764b0ef0544a0366be6cfb845d3141b411: Status 404 returned error can't find the container with id 4e763662b888a6a2fd4e9c24bafa4d764b0ef0544a0366be6cfb845d3141b411 Nov 02 07:37:49.414102 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.414074 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:37:49.414197 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.414133 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:37:49.414197 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.414178 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:37:49.527839 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.527804 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" event={"ID":"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e","Type":"ContainerStarted","Data":"fbb38da793d95bcb5ebfddbd2a446afa0d328e3020ee29f4dae1d88d5170936b"} Nov 02 07:37:49.528042 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.527867 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" event={"ID":"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e","Type":"ContainerStarted","Data":"4e763662b888a6a2fd4e9c24bafa4d764b0ef0544a0366be6cfb845d3141b411"} Nov 02 07:37:49.529609 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.529581 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" event={"ID":"e43d749b-223f-4a0a-997a-a0637ddf2d00","Type":"ContainerStarted","Data":"8b64fdc2e4439c024aaeff20c56c5f26d5e8f0b361e92aa5cfc235dc0e9559ae"} Nov 02 07:37:49.529725 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.529631 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" event={"ID":"e43d749b-223f-4a0a-997a-a0637ddf2d00","Type":"ContainerStarted","Data":"a57d154d3261e326487e6ca073e2aface619e6e971f729f6eefcb39a55e58283"} Nov 02 07:37:49.573923 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.573856 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" podStartSLOduration=1.573841931 podStartE2EDuration="1.573841931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:37:49.572927556 +0000 UTC m=+769.871208395" watchObservedRunningTime="2026-11-02 07:37:49.573841931 +0000 UTC m=+769.872122839" Nov 02 07:37:49.620292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.620230 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" podStartSLOduration=1.620210745 podStartE2EDuration="1.620210745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:37:49.617625137 +0000 UTC m=+769.915905965" watchObservedRunningTime="2026-11-02 07:37:49.620210745 +0000 UTC m=+769.918491573" Nov 02 07:37:49.720875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:49.720784 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:50.099183 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:50.098580 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:50.106184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:50.101189 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 07:37:50.106184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:50.101251 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" podUID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 07:37:50.116417 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:50.116365 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:50.472239 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:50.463244 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:50.472239 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:50.468006 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7xqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 07:37:50.472239 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:50.468059 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" podUID="e43d749b-223f-4a0a-997a-a0637ddf2d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 07:37:51.093686 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:51.093650 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 07:37:51.094002 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:51.093958 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" podUID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 07:37:51.462725 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:51.462619 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7xqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 07:37:51.463407 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:51.463261 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" podUID="e43d749b-223f-4a0a-997a-a0637ddf2d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 07:37:52.025088 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:52.025053 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:52.101616 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:52.101573 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 07:37:52.101938 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:52.101896 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" podUID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 07:37:52.475241 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:52.474570 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7xqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 07:37:52.475241 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:52.474634 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" podUID="e43d749b-223f-4a0a-997a-a0637ddf2d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 07:37:53.091588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:53.091547 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 07:37:53.091922 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:53.091878 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" podUID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 07:37:53.462813 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:53.462691 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7xqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 07:37:53.463105 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:53.463053 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" podUID="e43d749b-223f-4a0a-997a-a0637ddf2d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 07:37:54.083717 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:54.083671 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:54.091514 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:54.091482 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 07:37:54.091763 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:54.091736 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" podUID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 07:37:54.468619 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:54.468524 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7xqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 07:37:54.468865 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:54.468836 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" podUID="e43d749b-223f-4a0a-997a-a0637ddf2d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 07:37:54.894851 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:54.894717 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:55.102184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:55.094662 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 07:37:55.102184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:55.094756 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" podUID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 07:37:55.465912 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:55.465320 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7xqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 07:37:55.465912 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:55.465380 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" podUID="e43d749b-223f-4a0a-997a-a0637ddf2d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 07:37:55.509637 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:55.509599 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:56.091756 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:56.091719 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 07:37:56.091945 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:56.091785 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" podUID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 07:37:56.172954 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:56.172868 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:56.462839 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:56.462738 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7xqxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 07:37:56.462839 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:56.462793 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" podUID="e43d749b-223f-4a0a-997a-a0637ddf2d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 07:37:57.096884 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:57.096852 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:57.097200 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:57.097150 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:57.097847 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:57.097827 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 07:37:57.913413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:57.913379 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:57.913824 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:57.913688 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:57.914534 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:57.914514 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" Nov 02 07:37:58.103237 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:58.103201 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:37:59.904569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:37:59.904527 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:00.074107 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.074070 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq"] Nov 02 07:38:00.078259 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.078232 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.105706 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.105667 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq"] Nov 02 07:38:00.173982 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.173884 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.173982 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.173931 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.173982 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.173958 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36051c8-200c-49f8-b02b-652bc857cedc-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.174300 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.174113 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.174300 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.174269 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.174387 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.174370 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.174432 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.174399 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq7kh\" (UniqueName: \"kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-kube-api-access-qq7kh\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.174501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.174487 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36051c8-200c-49f8-b02b-652bc857cedc-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.174545 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.174510 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.195951 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.195923 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0fbc17-openshift-default-dockercfg-p76f6\"" Nov 02 07:38:00.275727 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.275688 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.275908 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.275743 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.275908 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.275760 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq7kh\" (UniqueName: \"kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-kube-api-access-qq7kh\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.275908 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.275794 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36051c8-200c-49f8-b02b-652bc857cedc-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.275908 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.275812 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.275908 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.275844 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.276192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.275997 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.276192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.276035 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36051c8-200c-49f8-b02b-652bc857cedc-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.276192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.276079 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.276380 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.276205 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.276380 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.276260 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.276501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.276475 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.276558 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.276509 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.276830 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.276803 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36051c8-200c-49f8-b02b-652bc857cedc-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.278244 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.278220 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36051c8-200c-49f8-b02b-652bc857cedc-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.278356 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.278317 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.298972 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.298945 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.299184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.299146 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq7kh\" (UniqueName: \"kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-kube-api-access-qq7kh\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.397205 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.397140 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:00.712155 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.712132 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq"] Nov 02 07:38:00.714345 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:38:00.714318 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36051c8_200c_49f8_b02b_652bc857cedc.slice/crio-fb893f3708aeb5b3750e7cdfc6609d00dac1a307abbab63f9ddcb4b43cc43247 WatchSource:0}: Error finding container fb893f3708aeb5b3750e7cdfc6609d00dac1a307abbab63f9ddcb4b43cc43247: Status 404 returned error can't find the container with id fb893f3708aeb5b3750e7cdfc6609d00dac1a307abbab63f9ddcb4b43cc43247 Nov 02 07:38:00.716567 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.716538 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:38:00.716628 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.716599 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:38:00.716661 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:00.716638 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:38:01.130658 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:01.130616 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:01.715633 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:01.715598 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" event={"ID":"c36051c8-200c-49f8-b02b-652bc857cedc","Type":"ContainerStarted","Data":"563ec7325e06a124cd56b7707a7c0a0965788eadac48f930989275c19e1065e1"} Nov 02 07:38:01.715808 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:01.715651 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" event={"ID":"c36051c8-200c-49f8-b02b-652bc857cedc","Type":"ContainerStarted","Data":"fb893f3708aeb5b3750e7cdfc6609d00dac1a307abbab63f9ddcb4b43cc43247"} Nov 02 07:38:01.743679 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:01.743650 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:01.792208 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:01.788725 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" podStartSLOduration=1.788707022 podStartE2EDuration="1.788707022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:38:01.774709315 +0000 UTC m=+782.072990142" watchObservedRunningTime="2026-11-02 07:38:01.788707022 +0000 UTC m=+782.086987848" Nov 02 07:38:02.397732 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:02.397696 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:02.399516 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:02.399491 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 07:38:02.399644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:02.399537 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" podUID="c36051c8-200c-49f8-b02b-652bc857cedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 07:38:03.397841 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:03.397804 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 07:38:03.398273 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:03.397871 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" podUID="c36051c8-200c-49f8-b02b-652bc857cedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 07:38:03.581791 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:03.581748 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:04.397713 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:04.397673 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 07:38:04.397931 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:04.397736 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" podUID="c36051c8-200c-49f8-b02b-652bc857cedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 07:38:05.400188 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:05.397677 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 07:38:05.400188 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:05.397737 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" podUID="c36051c8-200c-49f8-b02b-652bc857cedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 07:38:05.421359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:05.421327 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:06.398613 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:06.398575 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 07:38:06.398822 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:06.398645 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" podUID="c36051c8-200c-49f8-b02b-652bc857cedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 07:38:07.398184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.397618 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 07:38:07.398184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.397685 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" podUID="c36051c8-200c-49f8-b02b-652bc857cedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 07:38:07.673437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.673349 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl"] Nov 02 07:38:07.729712 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.729590 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:07.757203 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.753676 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl"] Nov 02 07:38:07.760751 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.757578 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.796315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.795818 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cln5l\" (UniqueName: \"kubernetes.io/projected/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-kube-api-access-cln5l\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.796315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.795872 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-dshm\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.796315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.795951 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tmp-dir\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.796315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.796014 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-home\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.796315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.796055 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-model-cache\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.796315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.796101 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tls-certs\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.829702 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.829401 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7e7a83\"/\"openshift-service-ca.crt\"" Nov 02 07:38:07.829702 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.829530 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7e7a83\"/\"kube-root-ca.crt\"" Nov 02 07:38:07.842326 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.842282 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7e7a83\"/\"distinct-w1-7e7a83-kserve-self-signed-certs\"" Nov 02 07:38:07.846233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.843650 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7e7a83\"/\"default-dockercfg-z4w4q\"" Nov 02 07:38:07.938450 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.938256 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:07.963371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.950105 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cln5l\" (UniqueName: \"kubernetes.io/projected/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-kube-api-access-cln5l\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.963371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.950177 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-dshm\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.963371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.950268 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tmp-dir\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.963371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.950344 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-home\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.963371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.950387 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-model-cache\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.963371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.950432 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tls-certs\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.963371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.962644 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tmp-dir\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.964433 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.964395 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-model-cache\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:07.968561 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.968534 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-home\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:08.018405 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:07.988415 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-dshm\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:08.041864 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:08.041794 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cln5l\" (UniqueName: \"kubernetes.io/projected/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-kube-api-access-cln5l\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:08.071916 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:08.071872 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tls-certs\") pod \"distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:08.173675 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:08.173364 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:08.636189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:08.636134 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl"] Nov 02 07:38:08.638943 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:38:08.638909 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59c52c76_c947_49cc_bbf0_1ea2cf3834c5.slice/crio-e49b237ddcd32fa5e833622d4cec933a7d4b1496461a6421a0ad59c08fb5cd9b WatchSource:0}: Error finding container e49b237ddcd32fa5e833622d4cec933a7d4b1496461a6421a0ad59c08fb5cd9b: Status 404 returned error can't find the container with id e49b237ddcd32fa5e833622d4cec933a7d4b1496461a6421a0ad59c08fb5cd9b Nov 02 07:38:08.803970 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:08.803934 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" event={"ID":"59c52c76-c947-49cc-bbf0-1ea2cf3834c5","Type":"ContainerStarted","Data":"3c5c2b4d2dd2bdd898106d25a96b8b4697fe23eed9a2eb5fd283494885e6e187"} Nov 02 07:38:08.803970 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:08.803978 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" event={"ID":"59c52c76-c947-49cc-bbf0-1ea2cf3834c5","Type":"ContainerStarted","Data":"e49b237ddcd32fa5e833622d4cec933a7d4b1496461a6421a0ad59c08fb5cd9b"} Nov 02 07:38:08.846308 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:08.846243 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" podStartSLOduration=1.846226983 podStartE2EDuration="1.846226983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:38:08.844695844 +0000 UTC m=+789.142976669" watchObservedRunningTime="2026-11-02 07:38:08.846226983 +0000 UTC m=+789.144507807" Nov 02 07:38:08.925419 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:08.925220 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:08.926058 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:08.926033 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:08.926920 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:08.926879 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 07:38:09.174147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:09.174111 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:09.803547 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:09.803515 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:11.593261 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:11.593224 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:13.142554 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:38:13.142521 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:38:13.403911 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:13.403799 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:15.106002 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:15.105955 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:15.280086 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:15.280021 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:16.914274 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:16.914239 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:18.174664 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:18.174629 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:18.187885 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:18.187855 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:18.795267 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:18.795226 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:18.866786 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:18.866758 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:38:20.497596 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:20.497550 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:22.200309 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:22.200272 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:23.121356 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.121312 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4"] Nov 02 07:38:23.125941 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.125915 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.146358 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.146324 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4"] Nov 02 07:38:23.155122 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.155091 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-337764\"/\"e2e-conflict-b-d0b940c5-kserve-self-signed-certs\"" Nov 02 07:38:23.206136 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.206100 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:23.223399 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.223358 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-dshm\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.223563 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.223433 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-model-cache\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.223612 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.223576 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-tmp-dir\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.223666 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.223649 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5nqg\" (UniqueName: \"kubernetes.io/projected/89d72d47-4b51-4dba-9dc5-b081ec165957-kube-api-access-g5nqg\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.223766 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.223748 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-home\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.223812 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.223781 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d72d47-4b51-4dba-9dc5-b081ec165957-tls-certs\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.325020 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.324980 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-dshm\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.325246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.325059 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-model-cache\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.325246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.325122 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-tmp-dir\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.325763 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.325728 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5nqg\" (UniqueName: \"kubernetes.io/projected/89d72d47-4b51-4dba-9dc5-b081ec165957-kube-api-access-g5nqg\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.326355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.325958 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-model-cache\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.326472 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.326350 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-home\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.326472 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.326411 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d72d47-4b51-4dba-9dc5-b081ec165957-tls-certs\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.326586 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.326245 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-tmp-dir\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.326642 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.326584 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-home\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.328191 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.328132 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-dshm\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.330280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.330255 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d72d47-4b51-4dba-9dc5-b081ec165957-tls-certs\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.349079 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.349044 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5nqg\" (UniqueName: \"kubernetes.io/projected/89d72d47-4b51-4dba-9dc5-b081ec165957-kube-api-access-g5nqg\") pod \"e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.449543 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.449454 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:23.749210 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.749179 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4"] Nov 02 07:38:23.751582 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:38:23.751554 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89d72d47_4b51_4dba_9dc5_b081ec165957.slice/crio-b05559f6359bcb6eb09501e96ca8b5bca423565d547e67effe6b8656821267de WatchSource:0}: Error finding container b05559f6359bcb6eb09501e96ca8b5bca423565d547e67effe6b8656821267de: Status 404 returned error can't find the container with id b05559f6359bcb6eb09501e96ca8b5bca423565d547e67effe6b8656821267de Nov 02 07:38:23.884349 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.884309 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" event={"ID":"89d72d47-4b51-4dba-9dc5-b081ec165957","Type":"ContainerStarted","Data":"ec1ca1a9c5cf89e9ff414cf4703e5ca6d52c9ce147beef300f206e29dff6d961"} Nov 02 07:38:23.884518 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.884359 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" event={"ID":"89d72d47-4b51-4dba-9dc5-b081ec165957","Type":"ContainerStarted","Data":"b05559f6359bcb6eb09501e96ca8b5bca423565d547e67effe6b8656821267de"} Nov 02 07:38:23.927687 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:23.927636 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" podStartSLOduration=0.927621699 podStartE2EDuration="927.621699ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:38:23.924542702 +0000 UTC m=+804.222823526" watchObservedRunningTime="2026-11-02 07:38:23.927621699 +0000 UTC m=+804.225902523" Nov 02 07:38:24.008855 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:24.008775 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:24.450455 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:24.450377 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:25.819512 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:25.819454 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:27.598338 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:27.598306 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:29.431141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:29.431094 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:31.193386 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:31.193347 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:33.017142 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:33.016566 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:33.450392 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:33.450307 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:33.463558 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:33.463530 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:33.956302 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:33.956270 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:38:34.350390 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:34.350358 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:34.801358 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:34.801307 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:36.507331 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:36.507277 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:38.315863 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:38.315826 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:40.550221 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:40.550186 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:41.859363 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:41.859294 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:43.140812 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:38:43.140771 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:38:43.580230 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:43.580199 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:45.401377 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:45.401339 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:46.013703 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:46.013658 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:47.207146 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:47.207081 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:48.896653 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:48.896596 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:50.213422 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:50.213385 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4"] Nov 02 07:38:50.293781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:50.293747 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:50.719120 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:50.719079 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:52.608650 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:52.608608 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:54.404449 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:54.404414 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:56.105841 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:56.105809 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:57.916779 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:57.916749 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:58.516337 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:58.516286 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:38:59.821612 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:38:59.821535 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:01.437196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:01.437141 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:02.171996 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:02.171950 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:03.967447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:03.967415 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:05.248425 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.248389 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" podUID="89d72d47-4b51-4dba-9dc5-b081ec165957" containerName="main" containerID="cri-o://ec1ca1a9c5cf89e9ff414cf4703e5ca6d52c9ce147beef300f206e29dff6d961" gracePeriod=45 Nov 02 07:39:05.518848 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.518781 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:39:05.575031 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.574996 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:05.623139 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.623094 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-home\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-home\") pod \"89d72d47-4b51-4dba-9dc5-b081ec165957\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " Nov 02 07:39:05.623338 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.623220 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89d72d47-4b51-4dba-9dc5-b081ec165957-tls-certs\" (UniqueName: \"kubernetes.io/secret/89d72d47-4b51-4dba-9dc5-b081ec165957-tls-certs\") pod \"89d72d47-4b51-4dba-9dc5-b081ec165957\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " Nov 02 07:39:05.623338 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.623256 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-model-cache\") pod \"89d72d47-4b51-4dba-9dc5-b081ec165957\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " Nov 02 07:39:05.623461 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.623376 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-home" pod "89d72d47-4b51-4dba-9dc5-b081ec165957" (UID: "89d72d47-4b51-4dba-9dc5-b081ec165957"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:39:05.623461 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.623438 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-tmp-dir\") pod \"89d72d47-4b51-4dba-9dc5-b081ec165957\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " Nov 02 07:39:05.623572 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.623493 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d72d47-4b51-4dba-9dc5-b081ec165957-kube-api-access-g5nqg\" (UniqueName: \"kubernetes.io/projected/89d72d47-4b51-4dba-9dc5-b081ec165957-kube-api-access-g5nqg\") pod \"89d72d47-4b51-4dba-9dc5-b081ec165957\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " Nov 02 07:39:05.623629 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.623596 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-dshm\") pod \"89d72d47-4b51-4dba-9dc5-b081ec165957\" (UID: \"89d72d47-4b51-4dba-9dc5-b081ec165957\") " Nov 02 07:39:05.623684 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.623651 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-model-cache" pod "89d72d47-4b51-4dba-9dc5-b081ec165957" (UID: "89d72d47-4b51-4dba-9dc5-b081ec165957"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:39:05.623738 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.623701 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-tmp-dir" pod "89d72d47-4b51-4dba-9dc5-b081ec165957" (UID: "89d72d47-4b51-4dba-9dc5-b081ec165957"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:39:05.624089 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.624066 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:39:05.624089 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.624091 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:39:05.624324 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.624105 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:39:05.625613 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.625591 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-dshm" pod "89d72d47-4b51-4dba-9dc5-b081ec165957" (UID: "89d72d47-4b51-4dba-9dc5-b081ec165957"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:39:05.625880 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.625866 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d72d47-4b51-4dba-9dc5-b081ec165957-kube-api-access-g5nqg" pod "89d72d47-4b51-4dba-9dc5-b081ec165957" (UID: "89d72d47-4b51-4dba-9dc5-b081ec165957"). InnerVolumeSpecName "kube-api-access-g5nqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:39:05.626302 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.626275 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d72d47-4b51-4dba-9dc5-b081ec165957-tls-certs" pod "89d72d47-4b51-4dba-9dc5-b081ec165957" (UID: "89d72d47-4b51-4dba-9dc5-b081ec165957"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:39:05.720289 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.720258 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:05.724655 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.724629 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d72d47-4b51-4dba-9dc5-b081ec165957-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:39:05.724655 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.724655 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d72d47-4b51-4dba-9dc5-b081ec165957-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:39:05.724809 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.724664 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5nqg\" (UniqueName: \"kubernetes.io/projected/89d72d47-4b51-4dba-9dc5-b081ec165957-kube-api-access-g5nqg\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:39:05.856544 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:05.856511 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:06.161792 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:06.161712 5899 generic.go:356] "Generic (PLEG): container finished" podID="89d72d47-4b51-4dba-9dc5-b081ec165957" containerID="ec1ca1a9c5cf89e9ff414cf4703e5ca6d52c9ce147beef300f206e29dff6d961" exitCode=0 Nov 02 07:39:06.161792 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:06.161753 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" Nov 02 07:39:06.161988 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:06.161793 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" event={"ID":"89d72d47-4b51-4dba-9dc5-b081ec165957","Type":"ContainerDied","Data":"ec1ca1a9c5cf89e9ff414cf4703e5ca6d52c9ce147beef300f206e29dff6d961"} Nov 02 07:39:06.161988 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:06.161837 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4" event={"ID":"89d72d47-4b51-4dba-9dc5-b081ec165957","Type":"ContainerDied","Data":"b05559f6359bcb6eb09501e96ca8b5bca423565d547e67effe6b8656821267de"} Nov 02 07:39:06.161988 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:06.161855 5899 scope.go:122] "RemoveContainer" containerID="ec1ca1a9c5cf89e9ff414cf4703e5ca6d52c9ce147beef300f206e29dff6d961" Nov 02 07:39:06.171155 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:06.171135 5899 scope.go:122] "RemoveContainer" containerID="ec1ca1a9c5cf89e9ff414cf4703e5ca6d52c9ce147beef300f206e29dff6d961" Nov 02 07:39:06.171433 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:39:06.171414 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1ca1a9c5cf89e9ff414cf4703e5ca6d52c9ce147beef300f206e29dff6d961\": container with ID starting with ec1ca1a9c5cf89e9ff414cf4703e5ca6d52c9ce147beef300f206e29dff6d961 not found: ID does not exist" containerID="ec1ca1a9c5cf89e9ff414cf4703e5ca6d52c9ce147beef300f206e29dff6d961" Nov 02 07:39:06.171483 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:06.171459 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1ca1a9c5cf89e9ff414cf4703e5ca6d52c9ce147beef300f206e29dff6d961"} err="failed to get container status \"ec1ca1a9c5cf89e9ff414cf4703e5ca6d52c9ce147beef300f206e29dff6d961\": rpc error: code = NotFound desc = could not find container \"ec1ca1a9c5cf89e9ff414cf4703e5ca6d52c9ce147beef300f206e29dff6d961\": container with ID starting with ec1ca1a9c5cf89e9ff414cf4703e5ca6d52c9ce147beef300f206e29dff6d961 not found: ID does not exist" Nov 02 07:39:06.210781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:06.210755 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4"] Nov 02 07:39:06.222492 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:06.222465 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-337764/e2e-conflict-b-d0b940c5-kserve-7864bd6b4b-gnzs4"] Nov 02 07:39:07.737972 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:07.737937 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:07.818580 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:07.818547 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d72d47-4b51-4dba-9dc5-b081ec165957" path="/var/lib/kubelet/pods/89d72d47-4b51-4dba-9dc5-b081ec165957/volumes" Nov 02 07:39:09.493049 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:09.493013 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:11.289717 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.289683 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:11.898052 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.898021 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92"] Nov 02 07:39:11.898483 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.898470 5899 cpu_manager.go:414] "containerMap: removing container" podUID="89d72d47-4b51-4dba-9dc5-b081ec165957" containerName="main" Nov 02 07:39:11.898538 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.898484 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d72d47-4b51-4dba-9dc5-b081ec165957" containerName="main" Nov 02 07:39:11.898598 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.898589 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d72d47-4b51-4dba-9dc5-b081ec165957" containerName="main" Nov 02 07:39:11.903136 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.903116 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:11.919331 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.919310 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7e7a83-openshift-default-dockercfg-8bmgw\"" Nov 02 07:39:11.931808 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.931780 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92"] Nov 02 07:39:11.991675 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.991636 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:11.991852 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.991695 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:11.991852 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.991817 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:11.991852 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.991847 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf4gg\" (UniqueName: \"kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-kube-api-access-qf4gg\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:11.991965 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.991910 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:11.992014 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.991995 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:11.992049 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.992025 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:11.992141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.992124 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:11.992303 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:11.992282 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.093010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.092973 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.093010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.093015 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.093320 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.093054 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.093320 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.093147 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.093320 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.093222 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.093320 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.093265 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.093320 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.093309 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.093590 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.093343 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf4gg\" (UniqueName: \"kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-kube-api-access-qf4gg\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.093590 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.093395 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.093590 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.093517 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.093590 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.093523 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.093963 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.093943 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.094039 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.093972 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.094039 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.094027 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.095664 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.095640 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.095859 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.095841 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.115894 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.115864 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.116012 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.115996 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf4gg\" (UniqueName: \"kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-kube-api-access-qf4gg\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.217402 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.217324 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:12.531811 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.531784 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92"] Nov 02 07:39:12.533643 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:39:12.533615 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cefcf5_4b43_4286_8e0f_7bf2b228eeb9.slice/crio-5959a8a813b085cd7c89363d020bf738d45038bb8e9c08e7bfff8c23c42c1592 WatchSource:0}: Error finding container 5959a8a813b085cd7c89363d020bf738d45038bb8e9c08e7bfff8c23c42c1592: Status 404 returned error can't find the container with id 5959a8a813b085cd7c89363d020bf738d45038bb8e9c08e7bfff8c23c42c1592 Nov 02 07:39:12.535934 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.535903 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:39:12.536015 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.535966 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:39:12.536015 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:12.535993 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:39:13.003306 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:13.003264 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:13.163342 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:39:13.163301 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:39:13.204179 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:13.204125 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" event={"ID":"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9","Type":"ContainerStarted","Data":"5a9a00d25b69a299c05c9837d2ee70f8ef3c8a72b6fdba3bdceb84627fc13496"} Nov 02 07:39:13.204359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:13.204209 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" event={"ID":"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9","Type":"ContainerStarted","Data":"5959a8a813b085cd7c89363d020bf738d45038bb8e9c08e7bfff8c23c42c1592"} Nov 02 07:39:13.217778 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:13.217750 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:13.219493 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:13.219470 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 07:39:13.219593 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:13.219545 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 07:39:13.269121 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:13.268993 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" podStartSLOduration=2.268970877 podStartE2EDuration="2.268970877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:13.262613228 +0000 UTC m=+853.560894054" watchObservedRunningTime="2026-11-02 07:39:13.268970877 +0000 UTC m=+853.567251703" Nov 02 07:39:14.022259 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:14.022207 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:14.218757 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:14.218722 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 07:39:14.218942 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:14.218785 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 07:39:14.786081 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:14.786040 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:15.217963 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.217890 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 07:39:15.217963 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.217936 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 07:39:15.347422 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.347374 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47"] Nov 02 07:39:15.351280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.351259 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.392468 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.392435 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47"] Nov 02 07:39:15.530445 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.530408 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.530445 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.530447 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.530662 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.530476 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzxh\" (UniqueName: \"kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-kube-api-access-6kzxh\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.530662 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.530602 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.530662 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.530634 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.530662 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.530656 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.530799 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.530739 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.530836 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.530805 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.530924 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.530901 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.632499 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.632460 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.632680 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.632521 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.632680 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.632594 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.632680 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.632621 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.632874 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.632756 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzxh\" (UniqueName: \"kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-kube-api-access-6kzxh\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.632874 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.632846 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.632977 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.632874 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.632977 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.632899 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.632977 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.632917 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.632977 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.632951 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.633192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.633062 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.633257 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.633194 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.633296 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.633254 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.633602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.633585 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.634881 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.634858 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.635234 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.635217 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.680753 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.680712 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.680903 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.680715 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzxh\" (UniqueName: \"kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-kube-api-access-6kzxh\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:15.693520 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:15.693495 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:16.014910 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:16.014866 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47"] Nov 02 07:39:16.020054 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:16.020011 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:39:16.020218 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:16.020148 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:39:16.020434 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:16.020402 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:39:16.218635 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:16.218570 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 07:39:16.219115 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:16.218634 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 07:39:16.232492 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:16.231183 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" event={"ID":"e5cde1bc-a874-4ec0-b40c-c294de1a54bf","Type":"ContainerStarted","Data":"e89ec449e80d850230751a585f5499bec9b99d88433ac0a084209ea51ff9617e"} Nov 02 07:39:16.232492 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:16.231276 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" event={"ID":"e5cde1bc-a874-4ec0-b40c-c294de1a54bf","Type":"ContainerStarted","Data":"65f129325815196dfc2ea41e82079e6f7d21b7a3811deb78c1d5b410c62fcb70"} Nov 02 07:39:16.704186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:16.703313 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:17.220191 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:17.219676 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 07:39:17.220191 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:17.219757 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 07:39:17.561184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:17.560848 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:17.699048 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:17.699004 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 07:39:17.699371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:17.699325 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 07:39:18.224189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:18.220494 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 07:39:18.224189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:18.220556 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 07:39:18.695586 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:18.695545 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 07:39:18.695858 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:18.695832 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 07:39:19.016087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:19.016036 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:39:19.143285 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:19.143240 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:39:19.181324 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:19.181271 5899 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:39:19.181898 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:19.181867 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" podUID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:39:19.220613 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:19.220571 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 07:39:19.220868 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:19.220837 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 07:39:19.698185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:19.697759 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 07:39:19.698185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:19.697846 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 07:39:20.031185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:20.028960 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:39:20.047188 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:20.046210 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:39:20.067320 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:20.067257 5899 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:39:20.200231 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:20.167124 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" podStartSLOduration=5.1671036 podStartE2EDuration="5.1671036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:16.286394665 +0000 UTC m=+856.584675502" watchObservedRunningTime="2026-11-02 07:39:20.1671036 +0000 UTC m=+860.465384427" Nov 02 07:39:20.202398 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:20.202368 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:39:20.237187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:20.218342 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 07:39:20.237455 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:20.237418 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 07:39:20.625523 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:20.625468 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:20.694767 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:20.694726 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 07:39:20.694920 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:20.694788 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 07:39:21.488741 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:21.488705 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:21.490170 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:21.490133 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:21.491449 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:21.491421 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 07:39:21.694292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:21.694255 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 07:39:21.694460 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:21.694311 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 07:39:22.649753 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:22.649718 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:22.694881 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:22.694845 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 07:39:22.695067 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:22.694911 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 07:39:23.694847 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:23.694813 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 07:39:23.695266 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:23.694864 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 07:39:24.491693 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:24.491652 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:24.694618 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:24.694570 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 07:39:24.694793 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:24.694642 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 07:39:25.699360 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:25.699328 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:25.699858 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:25.699614 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:25.700293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:25.700274 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 07:39:26.418821 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:26.418787 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:28.208273 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:28.208234 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:30.224563 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:30.224435 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:32.717182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:32.717120 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:33.188459 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.188424 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx"] Nov 02 07:39:33.196435 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.196402 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.211999 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.211969 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx"] Nov 02 07:39:33.220099 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.220069 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-354660\"/\"openshift-service-ca.crt\"" Nov 02 07:39:33.220360 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.220319 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-354660\"/\"kube-root-ca.crt\"" Nov 02 07:39:33.222638 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.222610 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-354660\"/\"distinct-w3-354660-kserve-self-signed-certs\"" Nov 02 07:39:33.222638 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.222610 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-354660\"/\"default-dockercfg-kkdsp\"" Nov 02 07:39:33.295394 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.295364 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:33.318755 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.318720 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-home\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.318923 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.318784 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-tmp-dir\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.318923 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.318892 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-dshm\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.319014 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.318945 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-model-cache\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.319014 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.319003 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swxks\" (UniqueName: \"kubernetes.io/projected/908ba798-56ae-457f-a348-88ae00e00ed9-kube-api-access-swxks\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.319086 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.319036 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908ba798-56ae-457f-a348-88ae00e00ed9-tls-certs\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.420537 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.420496 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-dshm\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.420786 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.420608 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-model-cache\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.420786 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.420641 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swxks\" (UniqueName: \"kubernetes.io/projected/908ba798-56ae-457f-a348-88ae00e00ed9-kube-api-access-swxks\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.420786 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.420723 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908ba798-56ae-457f-a348-88ae00e00ed9-tls-certs\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.421121 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.420835 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-home\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.421121 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.420911 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-tmp-dir\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.421377 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.421350 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-tmp-dir\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.421649 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.421623 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-home\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.421782 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.421717 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-model-cache\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.424482 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.424450 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-dshm\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.424865 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.424843 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908ba798-56ae-457f-a348-88ae00e00ed9-tls-certs\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.443943 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.443875 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swxks\" (UniqueName: \"kubernetes.io/projected/908ba798-56ae-457f-a348-88ae00e00ed9-kube-api-access-swxks\") pod \"distinct-w3-354660-kserve-547796d646-hkctx\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.513297 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.513260 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:33.820335 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:33.820312 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx"] Nov 02 07:39:33.822275 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:39:33.822250 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod908ba798_56ae_457f_a348_88ae00e00ed9.slice/crio-baa84e83a28b1a81c22ba2ae57cb301982543be174f51dacfa94bcadd535719b WatchSource:0}: Error finding container baa84e83a28b1a81c22ba2ae57cb301982543be174f51dacfa94bcadd535719b: Status 404 returned error can't find the container with id baa84e83a28b1a81c22ba2ae57cb301982543be174f51dacfa94bcadd535719b Nov 02 07:39:34.540644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:34.540606 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" event={"ID":"908ba798-56ae-457f-a348-88ae00e00ed9","Type":"ContainerStarted","Data":"9ebe781a811a12d313a17b15991df29062078ba33f53f4fbe48ed41e7c11ffa6"} Nov 02 07:39:34.540886 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:34.540656 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" event={"ID":"908ba798-56ae-457f-a348-88ae00e00ed9","Type":"ContainerStarted","Data":"baa84e83a28b1a81c22ba2ae57cb301982543be174f51dacfa94bcadd535719b"} Nov 02 07:39:34.581045 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:34.580993 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" podStartSLOduration=1.5809812920000001 podStartE2EDuration="1.580981292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:34.580599746 +0000 UTC m=+874.878880572" watchObservedRunningTime="2026-11-02 07:39:34.580981292 +0000 UTC m=+874.879262116" Nov 02 07:39:34.890314 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:34.890222 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:35.513657 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:35.513610 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:36.851428 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:36.851389 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:38.729044 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:38.729013 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:40.508484 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:40.508442 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:42.302327 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:42.302293 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:43.143383 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:39:43.143343 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:39:43.514522 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:43.514471 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:43.528791 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:43.528756 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:43.607515 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:43.607480 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:39:44.710184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:44.699796 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:45.066705 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.066658 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw"] Nov 02 07:39:45.070575 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.070550 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.099767 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.099736 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-337764\"/\"distinct-2-w2-337764-kserve-self-signed-certs\"" Nov 02 07:39:45.101905 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.101880 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw"] Nov 02 07:39:45.151704 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.151668 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:45.177765 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.177723 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-model-cache\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.177965 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.177874 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw8ld\" (UniqueName: \"kubernetes.io/projected/3f6a95fe-287c-4039-9165-6464f1a757e0-kube-api-access-zw8ld\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.177965 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.177907 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-tmp-dir\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.177965 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.177952 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-dshm\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.178142 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.178027 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6a95fe-287c-4039-9165-6464f1a757e0-tls-certs\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.178142 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.178079 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-home\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.278811 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.278771 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-model-cache\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.279014 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.278893 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw8ld\" (UniqueName: \"kubernetes.io/projected/3f6a95fe-287c-4039-9165-6464f1a757e0-kube-api-access-zw8ld\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.279014 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.278919 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-tmp-dir\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.279014 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.278991 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-dshm\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.279213 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.279027 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6a95fe-287c-4039-9165-6464f1a757e0-tls-certs\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.279213 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.279083 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-home\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.279341 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.279295 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-model-cache\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.279397 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.279375 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-tmp-dir\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.279601 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.279582 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-home\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.281638 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.281611 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6a95fe-287c-4039-9165-6464f1a757e0-tls-certs\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.281761 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.281621 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-dshm\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.302149 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.302116 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw8ld\" (UniqueName: \"kubernetes.io/projected/3f6a95fe-287c-4039-9165-6464f1a757e0-kube-api-access-zw8ld\") pod \"distinct-2-w2-337764-kserve-57f5c455f4-9cjfw\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.384788 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.384683 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:45.694198 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:45.694155 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw"] Nov 02 07:39:45.696247 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:39:45.696217 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f6a95fe_287c_4039_9165_6464f1a757e0.slice/crio-2cae60f7c9c99ac954c6e1369e1f2abf64458de9900c0ee2e6afd0993736157c WatchSource:0}: Error finding container 2cae60f7c9c99ac954c6e1369e1f2abf64458de9900c0ee2e6afd0993736157c: Status 404 returned error can't find the container with id 2cae60f7c9c99ac954c6e1369e1f2abf64458de9900c0ee2e6afd0993736157c Nov 02 07:39:46.620241 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:46.620192 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" event={"ID":"3f6a95fe-287c-4039-9165-6464f1a757e0","Type":"ContainerStarted","Data":"ceaf2f4719b7654a3dd387f2fb11a5d1e9a65653bd35e95a8f67f49fcb89d2b2"} Nov 02 07:39:46.620671 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:46.620251 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" event={"ID":"3f6a95fe-287c-4039-9165-6464f1a757e0","Type":"ContainerStarted","Data":"2cae60f7c9c99ac954c6e1369e1f2abf64458de9900c0ee2e6afd0993736157c"} Nov 02 07:39:46.662699 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:46.662636 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" podStartSLOduration=1.662622588 podStartE2EDuration="1.662622588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:46.660414819 +0000 UTC m=+886.958695647" watchObservedRunningTime="2026-11-02 07:39:46.662622588 +0000 UTC m=+886.960903412" Nov 02 07:39:46.928029 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:46.927953 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:47.385400 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:47.385360 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:48.917896 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:48.917859 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:49.116328 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.116285 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78"] Nov 02 07:39:49.120717 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.120694 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.154588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.154554 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78"] Nov 02 07:39:49.226293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.226210 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.226293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.226258 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9mtl\" (UniqueName: \"kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-kube-api-access-v9mtl\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.226501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.226407 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.226501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.226483 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34975296-bf4a-4c2c-a4b2-0719760298da-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.226577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.226546 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.226577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.226564 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34975296-bf4a-4c2c-a4b2-0719760298da-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.226653 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.226584 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.226653 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.226605 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.226755 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.226726 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.327590 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.327547 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.327590 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.327592 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34975296-bf4a-4c2c-a4b2-0719760298da-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.327855 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.327622 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.327855 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.327655 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.327855 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.327700 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.327855 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.327745 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.327855 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.327784 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9mtl\" (UniqueName: \"kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-kube-api-access-v9mtl\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.328103 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.327854 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.328103 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.327935 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34975296-bf4a-4c2c-a4b2-0719760298da-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.328103 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.328066 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.328289 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.328219 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.328289 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.328241 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.328451 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.328430 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.328683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.328661 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34975296-bf4a-4c2c-a4b2-0719760298da-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.330070 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.330049 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34975296-bf4a-4c2c-a4b2-0719760298da-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.330184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.330150 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.351606 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.351572 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9mtl\" (UniqueName: \"kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-kube-api-access-v9mtl\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.351606 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.351591 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-txc78\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.436153 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.436117 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:49.750713 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.750688 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78"] Nov 02 07:39:49.754092 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:39:49.754064 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34975296_bf4a_4c2c_a4b2_0719760298da.slice/crio-9f97c484b84daa3eeacb8b99d68f011d267cd0a4b219f3a25ded9925224b68a9 WatchSource:0}: Error finding container 9f97c484b84daa3eeacb8b99d68f011d267cd0a4b219f3a25ded9925224b68a9: Status 404 returned error can't find the container with id 9f97c484b84daa3eeacb8b99d68f011d267cd0a4b219f3a25ded9925224b68a9 Nov 02 07:39:49.756588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.756558 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:39:49.756673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.756620 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:39:49.756673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:49.756650 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:39:50.647706 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:50.647658 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" event={"ID":"34975296-bf4a-4c2c-a4b2-0719760298da","Type":"ContainerStarted","Data":"1ba82a12e60340b97a9811df94c2f6deb3f689052457c2f7af3c729033197312"} Nov 02 07:39:50.648098 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:50.647747 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" event={"ID":"34975296-bf4a-4c2c-a4b2-0719760298da","Type":"ContainerStarted","Data":"9f97c484b84daa3eeacb8b99d68f011d267cd0a4b219f3a25ded9925224b68a9"} Nov 02 07:39:50.677044 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:50.677008 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:50.716526 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:50.716474 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" podStartSLOduration=1.7164601990000001 podStartE2EDuration="1.716460199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:50.703657036 +0000 UTC m=+891.001937862" watchObservedRunningTime="2026-11-02 07:39:50.716460199 +0000 UTC m=+891.014741024" Nov 02 07:39:51.301205 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:51.298766 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:51.433191 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:51.433128 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:51.437132 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:51.437112 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:51.438816 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:51.438793 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-txc78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 07:39:51.438874 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:51.438838 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" podUID="34975296-bf4a-4c2c-a4b2-0719760298da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 07:39:52.436783 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:52.436746 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-txc78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 07:39:52.437174 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:52.436814 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" podUID="34975296-bf4a-4c2c-a4b2-0719760298da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 07:39:52.681876 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:52.681838 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:53.436435 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:53.436398 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-txc78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 07:39:53.436618 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:53.436467 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" podUID="34975296-bf4a-4c2c-a4b2-0719760298da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 07:39:54.437467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:54.437427 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-txc78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 07:39:54.437862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:54.437495 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" podUID="34975296-bf4a-4c2c-a4b2-0719760298da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 07:39:54.720347 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:54.720244 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:55.385878 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:55.385832 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:55.398663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:55.398632 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:55.437059 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:55.437026 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-txc78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 07:39:55.437251 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:55.437099 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" podUID="34975296-bf4a-4c2c-a4b2-0719760298da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 07:39:55.711525 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:55.711442 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:39:56.520392 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:56.520354 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:56.783707 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:56.783609 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:56.784646 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:56.784614 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:56.786461 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:56.786439 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:39:57.500392 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:57.500326 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:58.410834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:58.410799 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:59.017389 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.017353 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:59.234116 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.231007 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:39:59.367962 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.367875 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz"] Nov 02 07:39:59.372362 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.372342 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.387446 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.387421 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-354660-openshift-default-dockercfg-kgbn5\"" Nov 02 07:39:59.402825 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.402783 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz"] Nov 02 07:39:59.470643 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.470604 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9jg\" (UniqueName: \"kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-kube-api-access-hb9jg\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.471117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.470660 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.471117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.470805 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.471117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.470849 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.471117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.470952 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.471117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.471018 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.471117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.471081 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.471446 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.471119 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.471446 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.471223 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.572882 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.572835 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.573089 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.572906 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.573089 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.572952 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.573089 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.573011 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.573089 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.573085 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9jg\" (UniqueName: \"kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-kube-api-access-hb9jg\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.573365 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.573137 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.573365 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.573264 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.573365 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.573296 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.573365 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.573343 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.573527 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.573360 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.573527 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.573421 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.573906 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.573881 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.574020 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.573986 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.574215 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.574193 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.575712 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.575692 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.575959 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.575941 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.596404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.596375 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.596560 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.596488 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9jg\" (UniqueName: \"kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-kube-api-access-hb9jg\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:39:59.686152 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:39:59.686054 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:40:00.036062 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.036031 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz"] Nov 02 07:40:00.038338 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:40:00.038308 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12bb3bc_8ded_4004_a7c3_625003bf4e74.slice/crio-5fe3b7db6cba14fc8c06f8d0af9f596e47d50e68817e1dbfd9306b9c41493b97 WatchSource:0}: Error finding container 5fe3b7db6cba14fc8c06f8d0af9f596e47d50e68817e1dbfd9306b9c41493b97: Status 404 returned error can't find the container with id 5fe3b7db6cba14fc8c06f8d0af9f596e47d50e68817e1dbfd9306b9c41493b97 Nov 02 07:40:00.041494 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.041455 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:00.041577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.041552 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:00.041618 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.041602 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:00.526191 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.525129 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s"] Nov 02 07:40:00.574187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.573448 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s"] Nov 02 07:40:00.574187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.573581 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:00.908249 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.903761 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:00.908249 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.903847 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:00.908249 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.903900 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:00.908249 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.907315 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:00.938192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.929478 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:00.938192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.929546 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:00.938192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.929722 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:00.938192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.929991 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:00.938192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.930180 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2spp\" (UniqueName: \"kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-kube-api-access-g2spp\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:00.958387 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.958343 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" event={"ID":"e12bb3bc-8ded-4004-a7c3-625003bf4e74","Type":"ContainerStarted","Data":"e1642f1d69a26da67af668eaa142699cafaa07ecdb5bba36ab666bf0985c7fbc"} Nov 02 07:40:00.958669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:00.958645 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" event={"ID":"e12bb3bc-8ded-4004-a7c3-625003bf4e74","Type":"ContainerStarted","Data":"5fe3b7db6cba14fc8c06f8d0af9f596e47d50e68817e1dbfd9306b9c41493b97"} Nov 02 07:40:01.029189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.026660 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" podStartSLOduration=2.026640934 podStartE2EDuration="2.026640934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:01.025086135 +0000 UTC m=+901.323366962" watchObservedRunningTime="2026-11-02 07:40:01.026640934 +0000 UTC m=+901.324921760" Nov 02 07:40:01.035055 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.035015 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.035372 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.035351 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.035510 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.035493 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.035642 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.035626 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.035795 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.035779 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.035916 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.035898 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2spp\" (UniqueName: \"kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-kube-api-access-g2spp\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.036086 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.036070 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.036233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.036215 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.036364 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.036347 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.037401 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.037379 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.038136 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.038115 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.049813 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.049784 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.051503 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.051231 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.051657 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.051438 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.065842 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.065113 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.078277 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.077731 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2spp\" (UniqueName: \"kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-kube-api-access-g2spp\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.117185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.084665 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.117185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.084914 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.204239 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.204123 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:01.719184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.710831 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:01.778197 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.766594 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:40:01.845204 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.839137 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 07:40:01.845204 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:01.839238 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" podUID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 07:40:02.118186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:02.116439 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s"] Nov 02 07:40:02.218415 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:02.216692 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:02.218415 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:02.216785 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:02.218415 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:02.216833 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:02.417709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:02.417581 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:02.686477 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:02.686401 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 07:40:02.686477 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:02.686456 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" podUID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 07:40:03.089550 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:03.089491 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" event={"ID":"1ececa38-98c8-4d62-b4fc-1f06ab4e887c","Type":"ContainerStarted","Data":"80e640484c1d52411e4a2e0451d2f2abf2738587b4a25ec5068981fa25446eb4"} Nov 02 07:40:03.090004 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:03.089580 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" event={"ID":"1ececa38-98c8-4d62-b4fc-1f06ab4e887c","Type":"ContainerStarted","Data":"fd847513fb465c2465681c036534778d4ed51f3eb178f5ae45346e6c9a335e8d"} Nov 02 07:40:03.147294 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:03.147240 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" podStartSLOduration=3.147226225 podStartE2EDuration="3.147226225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:03.145132546 +0000 UTC m=+903.443413384" watchObservedRunningTime="2026-11-02 07:40:03.147226225 +0000 UTC m=+903.445507050" Nov 02 07:40:03.204827 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:03.204797 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:03.206583 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:03.206555 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 07:40:03.206716 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:03.206641 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" podUID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 07:40:03.689185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:03.688239 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 07:40:03.689185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:03.688303 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" podUID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 07:40:04.211574 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:04.211531 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 07:40:04.212109 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:04.211595 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" podUID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 07:40:04.687184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:04.686568 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 07:40:04.687184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:04.686643 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" podUID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 07:40:05.208189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:05.206611 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 07:40:05.208189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:05.206691 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" podUID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 07:40:05.478066 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:05.477992 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:05.690588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:05.690542 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 07:40:05.690889 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:05.690859 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" podUID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 07:40:06.205564 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:06.205525 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 07:40:06.205875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:06.205846 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" podUID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 07:40:06.687549 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:06.687510 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 07:40:06.688141 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:06.688110 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" podUID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 07:40:07.204802 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:07.204753 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 07:40:07.204992 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:07.204832 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" podUID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 07:40:07.637602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:07.637565 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:07.689631 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:07.689595 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 07:40:07.690280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:07.690243 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" podUID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 07:40:08.208577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:08.208540 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 07:40:08.208755 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:08.208601 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" podUID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 07:40:08.691375 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:08.691331 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:40:08.692048 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:08.692023 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:40:08.692810 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:08.692789 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 07:40:09.205450 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:09.205405 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 07:40:09.205638 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:09.205471 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" podUID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 07:40:09.655155 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:09.655109 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:10.208660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:10.208632 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:10.209153 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:10.208970 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:10.209892 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:10.209871 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:40:11.535991 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:11.535904 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:11.675200 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:11.675151 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:12.080050 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.080008 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt"] Nov 02 07:40:12.085120 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.085100 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.120801 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.120764 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt"] Nov 02 07:40:12.132589 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.132558 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.132751 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.132617 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.132751 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.132703 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljv4n\" (UniqueName: \"kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-kube-api-access-ljv4n\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.132751 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.132734 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.132873 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.132770 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.132943 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.132917 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.132991 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.132957 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.133055 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.133027 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.133094 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.133083 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.233788 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.233757 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.233788 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.233790 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.234035 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.233807 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.234035 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.233823 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.234035 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.233938 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.234253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.234054 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.234253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.234134 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljv4n\" (UniqueName: \"kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-kube-api-access-ljv4n\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.234253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.234216 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.234253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.234252 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.234485 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.234288 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.234485 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.234342 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.234637 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.234615 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.234739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.234719 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.234790 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.234721 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.236483 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.236460 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.236569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.236552 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.256894 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.256862 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.257017 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.256987 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljv4n\" (UniqueName: \"kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-kube-api-access-ljv4n\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.399854 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.399775 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:12.716174 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.715731 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt"] Nov 02 07:40:12.718476 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:40:12.718448 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88387240_8d67_4c5a_b8b2_940c0f6d2d0c.slice/crio-efc25db8b186c5412e38ca710b5aeeb499efee219549338e36cc6c85ceea13f9 WatchSource:0}: Error finding container efc25db8b186c5412e38ca710b5aeeb499efee219549338e36cc6c85ceea13f9: Status 404 returned error can't find the container with id efc25db8b186c5412e38ca710b5aeeb499efee219549338e36cc6c85ceea13f9 Nov 02 07:40:12.721386 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.721354 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:12.721465 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.721427 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:12.721465 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:12.721456 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:13.146128 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:40:13.146091 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:40:13.184183 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:13.183342 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" event={"ID":"88387240-8d67-4c5a-b8b2-940c0f6d2d0c","Type":"ContainerStarted","Data":"30b2629bc957ecd6a3007046cfb8397b5bb340e76c9b312a7d3384f9ca424afa"} Nov 02 07:40:13.184183 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:13.183408 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" event={"ID":"88387240-8d67-4c5a-b8b2-940c0f6d2d0c","Type":"ContainerStarted","Data":"efc25db8b186c5412e38ca710b5aeeb499efee219549338e36cc6c85ceea13f9"} Nov 02 07:40:13.252042 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:13.251990 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" podStartSLOduration=1.251977923 podStartE2EDuration="1.251977923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:13.240088339 +0000 UTC m=+913.538369164" watchObservedRunningTime="2026-11-02 07:40:13.251977923 +0000 UTC m=+913.550258800" Nov 02 07:40:13.400093 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:13.400002 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:13.401674 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:13.401649 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 07:40:13.401848 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:13.401693 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" podUID="88387240-8d67-4c5a-b8b2-940c0f6d2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 07:40:13.405389 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:13.405368 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:14.400779 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:14.400744 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 07:40:14.401197 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:14.400797 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" podUID="88387240-8d67-4c5a-b8b2-940c0f6d2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 07:40:15.209845 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:15.209811 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:15.400529 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:15.400492 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 07:40:15.400730 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:15.400547 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" podUID="88387240-8d67-4c5a-b8b2-940c0f6d2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 07:40:16.400735 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:16.400700 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 07:40:16.401112 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:16.400755 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" podUID="88387240-8d67-4c5a-b8b2-940c0f6d2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 07:40:17.019544 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:17.019482 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:17.400619 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:17.400534 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 07:40:17.400619 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:17.400590 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" podUID="88387240-8d67-4c5a-b8b2-940c0f6d2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 07:40:18.400913 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:18.400873 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 07:40:18.401351 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:18.400929 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" podUID="88387240-8d67-4c5a-b8b2-940c0f6d2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 07:40:18.780780 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:18.780748 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:19.196036 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.195949 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx"] Nov 02 07:40:19.200278 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.200253 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.226609 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.226582 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx"] Nov 02 07:40:19.333315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.333277 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.333315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.333313 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.333522 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.333350 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.333522 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.333481 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.333522 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.333502 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.333522 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.333519 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4hfl\" (UniqueName: \"kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-kube-api-access-s4hfl\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.333645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.333615 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.333691 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.333678 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.333726 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.333700 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.400448 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.400409 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 07:40:19.400624 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.400465 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" podUID="88387240-8d67-4c5a-b8b2-940c0f6d2d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 07:40:19.435188 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.435125 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.435645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.435253 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.435645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.435277 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.435645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.435298 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.435645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.435316 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.435645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.435346 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.435645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.435447 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.435645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.435470 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.435645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.435485 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4hfl\" (UniqueName: \"kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-kube-api-access-s4hfl\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.436075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.435691 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.436075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.435791 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.436075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.435877 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.436075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.436021 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.436075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.436037 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.437619 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.437590 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.437970 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.437953 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.458004 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.457942 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4hfl\" (UniqueName: \"kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-kube-api-access-s4hfl\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.458129 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.458103 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.518827 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.518782 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:19.848651 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.848624 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx"] Nov 02 07:40:19.851350 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:40:19.851322 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67cc3ce4_bd3c_426e_9f7b_0ca5ca7cc769.slice/crio-12c3fcec1f90a61c80652ff09a014adfcd43a257118e1839c395f03b8189894f WatchSource:0}: Error finding container 12c3fcec1f90a61c80652ff09a014adfcd43a257118e1839c395f03b8189894f: Status 404 returned error can't find the container with id 12c3fcec1f90a61c80652ff09a014adfcd43a257118e1839c395f03b8189894f Nov 02 07:40:19.854543 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.854504 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:19.854629 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.854596 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:19.854682 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:19.854650 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:20.253715 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:20.253675 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" event={"ID":"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769","Type":"ContainerStarted","Data":"89d12d477780c3826cd25cb1dc1bc614837c2ed15652cb30bfc61625932fa251"} Nov 02 07:40:20.253988 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:20.253964 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" event={"ID":"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769","Type":"ContainerStarted","Data":"12c3fcec1f90a61c80652ff09a014adfcd43a257118e1839c395f03b8189894f"} Nov 02 07:40:20.405266 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:20.400348 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" podStartSLOduration=1.400325216 podStartE2EDuration="1.400325216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:20.392317844 +0000 UTC m=+920.690598690" watchObservedRunningTime="2026-11-02 07:40:20.400325216 +0000 UTC m=+920.698606042" Nov 02 07:40:20.412705 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:20.412666 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:20.413692 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:20.413504 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:20.434464 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:20.434180 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:40:20.525469 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:20.525373 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:20.526102 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:20.526077 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 07:40:20.526376 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:20.526335 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 07:40:21.521899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:21.521619 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 07:40:21.521899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:21.521700 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 07:40:21.577762 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:21.577662 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:21.645349 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:21.644919 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:21.706240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:21.705250 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:21.863937 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:21.863833 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 07:40:22.193394 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:22.193290 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:22.194657 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:22.194626 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" podUID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:22.523030 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:22.522553 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 07:40:22.523030 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:22.522617 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 07:40:23.208324 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:23.208264 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 02 07:40:23.380189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:23.376322 5899 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:23.551211 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:23.530721 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 07:40:23.551793 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:23.551749 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 07:40:23.606058 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:23.606012 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:24.187309 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:24.187247 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" podUID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:24.546060 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:24.546021 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 07:40:24.546746 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:24.546714 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 07:40:24.579190 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:24.569148 5899 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:24.614499 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:24.614457 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:24.635377 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:24.634063 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:40:24.713683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:24.713639 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:24.885359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:24.885250 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:25.153233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:25.152369 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:40:25.520371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:25.520317 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 07:40:25.520560 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:25.520392 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 07:40:26.181195 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:26.163987 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:26.519685 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:26.519641 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 07:40:26.519837 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:26.519714 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 07:40:27.519471 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:27.519435 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 07:40:27.519853 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:27.519495 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 07:40:28.366640 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:28.366604 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:28.519785 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:28.519752 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 07:40:28.520197 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:28.519805 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 07:40:29.095493 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:29.095460 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:29.231029 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:29.230982 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:29.525169 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:29.525138 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:29.525950 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:29.525921 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:29.526719 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:29.526696 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:40:30.394867 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:30.394834 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:32.123391 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:32.123361 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:33.825245 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:33.825210 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:35.618261 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:35.618218 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:37.315647 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:37.315617 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:39.766936 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:39.766884 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:40.035283 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:40.035184 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:40.606767 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:40.606525 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:41.143319 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:41.143264 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:41.228317 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:41.228273 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 02 07:40:41.275341 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:41.275294 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:40:41.275568 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:41.275380 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" podUID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:41.538616 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:41.538275 5899 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" podUID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:41.540235 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:41.540146 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:41.863804 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:41.861801 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:42.189380 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:42.189275 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:42.189851 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:42.189435 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" podUID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 02 07:40:42.189851 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:42.189768 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:40:42.230290 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:42.230248 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:40:42.242180 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:42.242018 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:40:42.308944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:42.308914 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:40:42.454600 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:42.454521 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:40:43.140275 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:40:43.140237 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:40:44.015348 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.015313 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:44.517071 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.517033 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh"] Nov 02 07:40:44.522926 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.522901 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.547866 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.547832 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh"] Nov 02 07:40:44.642346 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.642306 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.642346 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.642344 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.642548 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.642443 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.642548 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.642473 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cfj6\" (UniqueName: \"kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-kube-api-access-6cfj6\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.642619 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.642568 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.642653 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.642615 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.642653 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.642641 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.642774 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.642761 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.642816 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.642801 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.744234 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.744188 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.744423 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.744255 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.744423 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.744307 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.744423 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.744330 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.744423 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.744395 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.744423 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.744417 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cfj6\" (UniqueName: \"kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-kube-api-access-6cfj6\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.744694 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.744496 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.744694 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.744530 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.744694 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.744569 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.744694 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.744622 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.744694 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.744643 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.744918 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.744870 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.744918 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.744908 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.745380 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.745361 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.746742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.746711 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.746862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.746811 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.767210 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.767126 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.767355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.767336 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cfj6\" (UniqueName: \"kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-kube-api-access-6cfj6\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:44.840206 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:44.840155 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:45.164708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:45.164676 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh"] Nov 02 07:40:45.166603 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:40:45.166580 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8c511f_15f3_4c8f_a3ad_ba854fd36fa8.slice/crio-851121003da019a99fddfc3c5beabdf62fcd2e3b3333e1cf0029ac9b1c52c4e2 WatchSource:0}: Error finding container 851121003da019a99fddfc3c5beabdf62fcd2e3b3333e1cf0029ac9b1c52c4e2: Status 404 returned error can't find the container with id 851121003da019a99fddfc3c5beabdf62fcd2e3b3333e1cf0029ac9b1c52c4e2 Nov 02 07:40:45.168904 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:45.168870 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:45.168997 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:45.168948 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:45.169066 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:45.169018 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:45.830242 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:45.830206 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" event={"ID":"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8","Type":"ContainerStarted","Data":"7a2592e89229dd181f51f0d64b85b9bb12283be0db710adab7ab48057decad08"} Nov 02 07:40:45.830457 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:45.830259 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" event={"ID":"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8","Type":"ContainerStarted","Data":"851121003da019a99fddfc3c5beabdf62fcd2e3b3333e1cf0029ac9b1c52c4e2"} Nov 02 07:40:45.840591 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:45.840556 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:45.841588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:45.841552 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:46.842713 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:46.842658 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 07:40:46.843257 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:46.842751 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" podUID="1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 07:40:47.182917 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:47.182808 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:47.324346 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:47.324305 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:47.841178 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:47.841129 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 07:40:47.841382 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:47.841190 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" podUID="1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 07:40:48.080837 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:48.080697 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:48.840917 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:48.840878 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 07:40:48.841096 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:48.840940 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" podUID="1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 07:40:48.985102 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:48.985063 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:49.136348 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:49.136217 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:49.841285 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:49.841253 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 07:40:49.841479 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:49.841319 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" podUID="1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 07:40:50.043663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:50.043546 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:50.840981 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:50.840943 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 07:40:50.841381 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:50.840998 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" podUID="1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 07:40:51.841395 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:51.841348 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 07:40:51.841794 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:51.841427 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" podUID="1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 07:40:52.001673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:52.001632 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:52.845738 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:52.845702 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:52.846269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:52.846124 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:52.846896 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:52.846877 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:40:52.915282 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:52.915221 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" podStartSLOduration=8.915202105 podStartE2EDuration="8.915202105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:45.894634478 +0000 UTC m=+946.192915304" watchObservedRunningTime="2026-11-02 07:40:52.915202105 +0000 UTC m=+953.213482932" Nov 02 07:40:53.794489 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:53.794459 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:55.518433 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:55.518395 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:56.752471 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:56.752435 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:56.897433 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:56.897386 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:57.037036 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.036949 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:57.240384 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.240352 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr"] Nov 02 07:40:57.244770 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.244750 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.285771 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.285737 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr"] Nov 02 07:40:57.389577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.389493 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:57.389755 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.389732 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.389803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.389777 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.389935 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.389912 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.389970 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.389958 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.390062 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.390046 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7dr8\" (UniqueName: \"kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-kube-api-access-m7dr8\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.390096 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.390077 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.390133 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.390105 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0257772-1251-45b3-8c95-ffbe53bda5f3-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.390253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.390239 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.390293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.390273 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.491814 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.491774 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.492019 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.491841 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.492019 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.491861 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.492019 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.491879 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7dr8\" (UniqueName: \"kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-kube-api-access-m7dr8\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.492019 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.491904 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.492019 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.491926 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0257772-1251-45b3-8c95-ffbe53bda5f3-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.492308 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.492055 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.492308 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.492093 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.492308 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.492261 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.492467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.492406 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.492467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.492433 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.492576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.492504 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.492641 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.492618 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.493029 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.493006 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0257772-1251-45b3-8c95-ffbe53bda5f3-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.494467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.494446 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.494840 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.494820 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.516405 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.516372 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.516858 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.516831 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7dr8\" (UniqueName: \"kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-kube-api-access-m7dr8\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.522641 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.522607 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:57.563832 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.563796 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:57.896911 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.896888 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr"] Nov 02 07:40:57.899881 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:40:57.899854 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0257772_1251_45b3_8c95_ffbe53bda5f3.slice/crio-a8a785e25d8b9efe6de28e03250efaf0b94e0ef75494459449b9da792eec4bd5 WatchSource:0}: Error finding container a8a785e25d8b9efe6de28e03250efaf0b94e0ef75494459449b9da792eec4bd5: Status 404 returned error can't find the container with id a8a785e25d8b9efe6de28e03250efaf0b94e0ef75494459449b9da792eec4bd5 Nov 02 07:40:57.902216 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.902184 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:57.902295 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.902247 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:57.902295 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.902275 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:40:57.913551 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.913525 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:57.935436 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:57.935396 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" event={"ID":"d0257772-1251-45b3-8c95-ffbe53bda5f3","Type":"ContainerStarted","Data":"a8a785e25d8b9efe6de28e03250efaf0b94e0ef75494459449b9da792eec4bd5"} Nov 02 07:40:58.051519 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:58.051482 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:58.192610 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:58.192523 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:58.945024 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:58.944989 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" event={"ID":"d0257772-1251-45b3-8c95-ffbe53bda5f3","Type":"ContainerStarted","Data":"4c2197862016cfc5808219da504f7670accd4ed1e3814b84f3ee0b2c63f4f3a0"} Nov 02 07:40:58.997197 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:58.997118 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" podStartSLOduration=1.99709588 podStartE2EDuration="1.99709588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:58.994988612 +0000 UTC m=+959.293269457" watchObservedRunningTime="2026-11-02 07:40:58.99709588 +0000 UTC m=+959.295376708" Nov 02 07:40:59.304277 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.304242 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz"] Nov 02 07:40:59.309337 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.309313 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.329203 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.329139 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz"] Nov 02 07:40:59.380181 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.379495 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9db802\"/\"distinct-2-w4-9db802-kserve-self-signed-certs\"" Nov 02 07:40:59.408362 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.408332 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:59.422967 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.422930 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-model-cache\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.423202 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.423000 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4wc9\" (UniqueName: \"kubernetes.io/projected/6725cf86-9321-401c-85b8-fba788e1e5c5-kube-api-access-f4wc9\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.423279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.423238 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-home\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.423344 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.423294 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-tmp-dir\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.423407 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.423382 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-dshm\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.423516 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.423496 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6725cf86-9321-401c-85b8-fba788e1e5c5-tls-certs\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.525256 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.525207 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-tmp-dir\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.525256 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.525255 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-dshm\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.525508 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.525334 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6725cf86-9321-401c-85b8-fba788e1e5c5-tls-certs\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.525508 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.525405 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-model-cache\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.525508 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.525442 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4wc9\" (UniqueName: \"kubernetes.io/projected/6725cf86-9321-401c-85b8-fba788e1e5c5-kube-api-access-f4wc9\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.525667 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.525531 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-home\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.525730 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.525703 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-tmp-dir\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.525858 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.525812 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-model-cache\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.525858 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.525855 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-home\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.527965 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.527917 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-dshm\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.528106 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.528089 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6725cf86-9321-401c-85b8-fba788e1e5c5-tls-certs\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.552205 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.549986 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:40:59.552205 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.550288 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4wc9\" (UniqueName: \"kubernetes.io/projected/6725cf86-9321-401c-85b8-fba788e1e5c5-kube-api-access-f4wc9\") pod \"distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.555007 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.554929 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn"] Nov 02 07:40:59.572578 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.571764 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 07:40:59.572578 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.571823 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 07:40:59.620418 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.614449 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:40:59.620418 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.614500 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn"] Nov 02 07:40:59.620418 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.614609 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.628391 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.627096 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:40:59.729966 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.729932 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.730147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.729999 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.730147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.730014 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.730147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.730042 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.730147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.730074 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.730147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.730111 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw96q\" (UniqueName: \"kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-kube-api-access-xw96q\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.730147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.730133 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.730380 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.730203 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.730380 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.730218 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.834633 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.833267 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw96q\" (UniqueName: \"kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-kube-api-access-xw96q\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.834633 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.833321 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.834633 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.833379 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.834633 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.833404 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.834633 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.833474 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.834633 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.833559 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.834633 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.833583 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.834633 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.833627 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.834633 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.833674 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.835330 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.835290 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.835718 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.835698 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.836479 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.836459 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.836748 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.836723 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.836855 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.836819 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.838121 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.838085 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.845178 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.845134 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.890739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.888309 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.890739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.888389 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw96q\" (UniqueName: \"kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-kube-api-access-xw96q\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:40:59.942838 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:40:59.942805 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:41:00.040490 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:00.039971 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz"] Nov 02 07:41:00.278182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:00.277844 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn"] Nov 02 07:41:00.284557 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:41:00.284518 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ca22de_6df7_4d64_a7d3_4aa4653ce6b8.slice/crio-4976c6e0c1ef1a198da3613b4846e6f0b0998f752bca7915d63ccc761535c671 WatchSource:0}: Error finding container 4976c6e0c1ef1a198da3613b4846e6f0b0998f752bca7915d63ccc761535c671: Status 404 returned error can't find the container with id 4976c6e0c1ef1a198da3613b4846e6f0b0998f752bca7915d63ccc761535c671 Nov 02 07:41:00.287252 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:00.287216 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:41:00.287336 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:00.287288 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:41:00.287336 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:00.287320 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:41:00.577183 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:00.576547 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 07:41:00.577183 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:00.576612 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 07:41:01.002520 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:01.001455 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" event={"ID":"6725cf86-9321-401c-85b8-fba788e1e5c5","Type":"ContainerStarted","Data":"3044a3b26c1c078c0d608965f519f7161b0e2677f3c755fbc8389d9282c1536f"} Nov 02 07:41:01.002520 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:01.001510 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" event={"ID":"6725cf86-9321-401c-85b8-fba788e1e5c5","Type":"ContainerStarted","Data":"72cd0a71e456719d40c7d83c485cbbbdb43942920bcf219c60903a3b874b1fcd"} Nov 02 07:41:01.016812 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:01.009298 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" event={"ID":"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8","Type":"ContainerStarted","Data":"90c6cd63f7fea360de8d4cb0a340eaaac868281840a1e8e0bcc876e375b9d8ae"} Nov 02 07:41:01.016812 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:01.009392 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" event={"ID":"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8","Type":"ContainerStarted","Data":"4976c6e0c1ef1a198da3613b4846e6f0b0998f752bca7915d63ccc761535c671"} Nov 02 07:41:01.112191 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:01.110874 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" podStartSLOduration=2.110854405 podStartE2EDuration="2.110854405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:01.05973876 +0000 UTC m=+961.358019587" watchObservedRunningTime="2026-11-02 07:41:01.110854405 +0000 UTC m=+961.409135232" Nov 02 07:41:01.127859 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:01.127788 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" podStartSLOduration=2.127768747 podStartE2EDuration="2.127768747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:01.108476772 +0000 UTC m=+961.406757599" watchObservedRunningTime="2026-11-02 07:41:01.127768747 +0000 UTC m=+961.426049573" Nov 02 07:41:01.440240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:01.435564 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:01.565645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:01.565603 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 07:41:01.565949 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:01.565918 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 07:41:01.643199 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:01.640257 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:41:01.981187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:01.980788 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:41:02.189644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:02.187154 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 07:41:02.189644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:02.187233 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 07:41:02.233918 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:02.233691 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:02.258651 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:02.258610 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:02.333350 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:02.333292 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:02.434210 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:02.410772 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:02.569390 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:02.569297 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:02.571076 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:02.571050 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 07:41:02.571279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:02.571255 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 07:41:02.711374 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:02.711316 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:02.856359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:02.856260 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:02.950599 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:02.950566 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 07:41:02.950889 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:02.950861 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 07:41:03.204241 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.195251 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:03.336210 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.335774 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:03.337076 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.336943 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:03.337325 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.337305 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:41:03.360293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.353963 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:41:03.389569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.378250 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:03.436076 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.436030 5899 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Nov 02 07:41:03.664188 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.645620 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 07:41:03.664188 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.645683 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 07:41:03.728282 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.726565 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 02 07:41:03.728282 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.727213 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:41:03.872981 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.872931 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:03.876613 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.876564 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:03.967174 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.964583 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 07:41:03.967174 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:03.964645 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 07:41:04.052682 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:04.050825 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:41:04.160220 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:04.159567 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 07:41:04.204273 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:04.202422 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:04.204273 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:04.203209 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:41:04.338185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:04.336003 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:04.364331 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:04.362254 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 07:41:04.642246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:04.569939 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 07:41:04.642246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:04.570344 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 07:41:04.889365 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:04.889311 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:04.945241 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:04.944878 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 07:41:04.945241 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:04.944966 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 07:41:05.165063 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:05.165028 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 07:41:05.187198 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:05.186218 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" podUID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:05.231356 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:05.211797 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:05.408184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:05.397312 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:05.680842 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:05.680753 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 07:41:05.681143 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:05.681112 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 07:41:05.693800 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:05.693758 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 07:41:05.912260 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:05.911342 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 07:41:05.950264 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:05.944554 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 07:41:05.950264 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:05.944619 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 07:41:05.953860 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:05.952715 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:41:06.175185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:06.175036 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:06.301622 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:06.301590 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 07:41:06.567626 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:06.567531 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 07:41:06.567881 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:06.567851 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 07:41:06.902235 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:06.900654 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp"] Nov 02 07:41:06.906107 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:06.906077 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:06.918932 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:06.918890 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp"] Nov 02 07:41:06.927324 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:06.927293 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9db802-openshift-default-dockercfg-4p9sf\"" Nov 02 07:41:06.944065 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:06.944024 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 07:41:06.944255 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:06.944092 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 07:41:06.992187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:06.991451 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:07.023991 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.023957 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.024212 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.024128 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.024212 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.024190 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.024343 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.024261 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.024343 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.024310 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.024445 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.024339 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l58gd\" (UniqueName: \"kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-kube-api-access-l58gd\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.024445 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.024385 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.024548 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.024464 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.024548 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.024502 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.126028 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.125974 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.126251 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.126072 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.126251 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.126111 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.126251 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.126214 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.126418 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.126262 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.126558 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.126534 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.126669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.126624 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.126746 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.126294 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58gd\" (UniqueName: \"kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-kube-api-access-l58gd\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.126818 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.126801 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.126931 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.126912 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.127023 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.127006 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.127215 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.127190 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.127783 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.127752 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.128140 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.128118 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.128563 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.128539 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.128956 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.128940 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.149665 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.149634 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.149817 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.149751 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58gd\" (UniqueName: \"kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-kube-api-access-l58gd\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.226201 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.226097 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:07.570647 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.570604 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 07:41:07.571300 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.571257 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 07:41:07.596729 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:41:07.596699 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3489f7c1_5a82_4ac7_8a71_7df03cd28051.slice/crio-351039132ec7fb1cd6709020a0e2a3e0ad3a979e545366a464f0dd6ad390e827 WatchSource:0}: Error finding container 351039132ec7fb1cd6709020a0e2a3e0ad3a979e545366a464f0dd6ad390e827: Status 404 returned error can't find the container with id 351039132ec7fb1cd6709020a0e2a3e0ad3a979e545366a464f0dd6ad390e827 Nov 02 07:41:07.596829 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.596789 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp"] Nov 02 07:41:07.599076 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.599045 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:41:07.599192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.599109 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:41:07.599192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.599139 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:41:07.730851 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.730808 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" event={"ID":"3489f7c1-5a82-4ac7-8a71-7df03cd28051","Type":"ContainerStarted","Data":"5ad83fd6d4435c1265fe17dfbde97ba0b88b0b140d336397c7e28a34919257ba"} Nov 02 07:41:07.731052 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.730883 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" event={"ID":"3489f7c1-5a82-4ac7-8a71-7df03cd28051","Type":"ContainerStarted","Data":"351039132ec7fb1cd6709020a0e2a3e0ad3a979e545366a464f0dd6ad390e827"} Nov 02 07:41:07.791185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.790459 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" podStartSLOduration=1.790430977 podStartE2EDuration="1.790430977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:07.782469081 +0000 UTC m=+968.080749919" watchObservedRunningTime="2026-11-02 07:41:07.790430977 +0000 UTC m=+968.088711802" Nov 02 07:41:07.943577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.943452 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 07:41:07.943577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:07.943523 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 07:41:08.235237 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:08.228495 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:08.235237 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:08.229778 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 07:41:08.235237 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:08.229827 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" podUID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 07:41:08.940214 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:08.940169 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:08.944125 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:08.944099 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 07:41:08.944285 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:08.944142 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 07:41:09.228127 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:09.226760 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 07:41:09.228127 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:09.226819 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" podUID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 07:41:09.598868 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:09.598780 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:09.630033 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:09.629994 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:41:09.641204 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:09.640261 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:41:09.641204 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:09.640325 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:09.651416 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:09.651393 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:41:09.737183 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:09.736816 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:09.806186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:09.803525 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 07:41:09.881670 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:09.881586 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:09.950599 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:09.950559 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 07:41:09.952001 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:09.951962 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 07:41:10.134259 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:10.134068 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:10.219986 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:10.219461 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:41:10.223408 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:10.220809 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:41:10.223408 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:10.223324 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:41:10.227889 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:10.227864 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 07:41:10.228002 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:10.227913 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" podUID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 07:41:10.415949 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:10.415821 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:10.733241 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:10.733101 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:10.943884 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:10.943845 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 07:41:10.944047 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:10.943913 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 07:41:11.229673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:11.229295 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 07:41:11.229673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:11.229351 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" podUID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 07:41:11.241556 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:11.241503 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:11.525653 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:11.525295 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:11.669072 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:11.669039 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:12.226862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:12.226821 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 07:41:12.227041 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:12.226878 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" podUID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 07:41:12.292521 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:12.292485 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:41:12.293860 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:12.293834 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:41:12.294911 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:12.294792 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 07:41:13.117612 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:13.117578 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:13.147647 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:41:13.147612 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:41:13.227215 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:13.227178 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 07:41:13.227470 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:13.227269 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" podUID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 07:41:14.226380 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:14.226350 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 07:41:14.226766 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:14.226399 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" podUID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 07:41:14.617382 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:14.617348 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:14.755704 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:14.755671 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:15.143265 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:15.143204 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:15.231215 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:15.231182 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:15.231702 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:15.231472 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:15.232345 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:15.232321 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 07:41:16.404042 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:16.404006 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:16.549077 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:16.549042 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:17.044885 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:17.044854 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:18.051357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:18.051322 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:20.205446 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:20.205402 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:22.477567 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:22.477525 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:22.828396 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:22.828296 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:24.217284 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:24.217238 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:25.239184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:25.218534 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:25.239184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:25.219277 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:41:25.240893 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:25.240050 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" podUID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:25.370185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:25.352073 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 07:41:25.730327 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:25.730286 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 02 07:41:25.943368 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:25.941226 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" podUID="6725cf86-9321-401c-85b8-fba788e1e5c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 07:41:26.192862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:26.192763 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:26.202043 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:26.201971 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 02 07:41:26.421184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:26.418661 5899 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Nov 02 07:41:27.034299 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.034260 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g"] Nov 02 07:41:27.043695 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.043661 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g"] Nov 02 07:41:27.044030 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.044007 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:27.128186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.083983 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d47676-serving-cert\"" Nov 02 07:41:27.137085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.137000 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:27.144177 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.144122 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tctvj\" (UniqueName: \"kubernetes.io/projected/b9077deb-a707-4161-b0bc-85899c58fca3-kube-api-access-tctvj\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:27.144409 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.144385 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-maas-api-tls\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:27.144665 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.144644 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-metrics-certs\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:27.144774 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.144753 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9077deb-a707-4161-b0bc-85899c58fca3-openshift-service-ca\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:27.263182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.248113 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-metrics-certs\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:27.263182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.251298 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9077deb-a707-4161-b0bc-85899c58fca3-openshift-service-ca\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:27.263182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.251375 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctvj\" (UniqueName: \"kubernetes.io/projected/b9077deb-a707-4161-b0bc-85899c58fca3-kube-api-access-tctvj\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:27.263182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.251410 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-maas-api-tls\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:27.263182 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:41:27.251611 5899 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-serving-cert: secret "maas-api-e2e-api-a-d47676-serving-cert" not found Nov 02 07:41:27.263182 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:41:27.251682 5899 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-maas-api-tls podName:b9077deb-a707-4161-b0bc-85899c58fca3 nodeName:}" failed. No retries permitted until 2026-11-02 07:41:27.751660097 +0000 UTC m=+988.049940902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-maas-api-tls") pod "maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" (UID: "b9077deb-a707-4161-b0bc-85899c58fca3") : secret "maas-api-e2e-api-a-d47676-serving-cert" not found Nov 02 07:41:27.263182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.252617 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9077deb-a707-4161-b0bc-85899c58fca3-openshift-service-ca\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:27.284815 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.284736 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-metrics-certs\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:27.326615 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.318067 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:27.326615 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.320111 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tctvj\" (UniqueName: \"kubernetes.io/projected/b9077deb-a707-4161-b0bc-85899c58fca3-kube-api-access-tctvj\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:27.494383 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.494339 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-d2sj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:41:27.495014 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.494976 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" podUID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:41:27.759546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:27.759509 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-maas-api-tls\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:27.759733 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:41:27.759700 5899 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-serving-cert: secret "maas-api-e2e-api-a-d47676-serving-cert" not found Nov 02 07:41:27.759831 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:41:27.759786 5899 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-maas-api-tls podName:b9077deb-a707-4161-b0bc-85899c58fca3 nodeName:}" failed. No retries permitted until 2026-11-02 07:41:28.759764059 +0000 UTC m=+989.058044881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-maas-api-tls") pod "maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" (UID: "b9077deb-a707-4161-b0bc-85899c58fca3") : secret "maas-api-e2e-api-a-d47676-serving-cert" not found Nov 02 07:41:28.207408 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:28.207318 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:28.772946 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:28.772900 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-maas-api-tls\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:28.775399 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:28.775368 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-maas-api-tls\") pod \"maas-api-e2e-api-a-d47676-8546fb879d-xzt8g\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:28.936591 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:28.936558 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:29.243851 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:29.243822 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g"] Nov 02 07:41:29.249690 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:29.249666 5899 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 07:41:29.603327 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:29.603287 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" event={"ID":"b9077deb-a707-4161-b0bc-85899c58fca3","Type":"ContainerStarted","Data":"2d290e3df39a138520f26c3ebd972fbf25edfcbf620f748af72f5d77cccbd052"} Nov 02 07:41:29.706557 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:29.706510 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:30.228204 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.228151 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff"] Nov 02 07:41:30.233981 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.232905 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.261057 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.261024 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-77dc65fb\"/\"default-dockercfg-4l9f5\"" Nov 02 07:41:30.261057 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.261039 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-77dc65fb\"/\"kube-root-ca.crt\"" Nov 02 07:41:30.261347 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.261088 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-77dc65fb\"/\"e2e-lifecycle-model-77dc65fb-kserve-self-signed-certs\"" Nov 02 07:41:30.261347 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.261144 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-77dc65fb\"/\"openshift-service-ca.crt\"" Nov 02 07:41:30.264665 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.264629 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff"] Nov 02 07:41:30.291770 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.291730 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-dshm\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.291770 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.291777 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-model-cache\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.292032 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.291807 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa74a97-484e-417c-9a5a-fb45c815b342-tls-certs\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.292032 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.291940 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-tmp-dir\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.292197 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.292178 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-home\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.292263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.292238 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbm5r\" (UniqueName: \"kubernetes.io/projected/faa74a97-484e-417c-9a5a-fb45c815b342-kube-api-access-qbm5r\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.318577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.318536 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:30.394009 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.393971 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-home\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.394200 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.394046 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbm5r\" (UniqueName: \"kubernetes.io/projected/faa74a97-484e-417c-9a5a-fb45c815b342-kube-api-access-qbm5r\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.394200 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.394125 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-dshm\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.394200 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.394153 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-model-cache\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.394365 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.394207 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa74a97-484e-417c-9a5a-fb45c815b342-tls-certs\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.394365 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.394235 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-tmp-dir\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.394517 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.394496 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-home\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.394934 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.394910 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-model-cache\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.395033 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.394982 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-tmp-dir\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.396660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.396638 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-dshm\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.396964 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.396937 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa74a97-484e-417c-9a5a-fb45c815b342-tls-certs\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.417111 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.417076 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbm5r\" (UniqueName: \"kubernetes.io/projected/faa74a97-484e-417c-9a5a-fb45c815b342-kube-api-access-qbm5r\") pod \"e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.549388 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.549287 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:30.612597 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.612560 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" event={"ID":"b9077deb-a707-4161-b0bc-85899c58fca3","Type":"ContainerStarted","Data":"27ab57106c0103abb4e56453d02d5e101c34225371fe3ff8af2eeba1113df5f7"} Nov 02 07:41:30.612769 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.612720 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:30.670274 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.668699 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" podStartSLOduration=4.375929357 podStartE2EDuration="4.668679812s" totalImagesPullingTime="292.750455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:30.664766051 +0000 UTC m=+990.963046901" watchObservedRunningTime="2026-11-02 07:41:30.668679812 +0000 UTC m=+990.966960639" Nov 02 07:41:30.871407 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:30.871309 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff"] Nov 02 07:41:30.876703 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:41:30.876666 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaa74a97_484e_417c_9a5a_fb45c815b342.slice/crio-1baaa38efa9d5e50dc1c4d80b1f7486509550c4706f65e1882b1dcf7ec38a131 WatchSource:0}: Error finding container 1baaa38efa9d5e50dc1c4d80b1f7486509550c4706f65e1882b1dcf7ec38a131: Status 404 returned error can't find the container with id 1baaa38efa9d5e50dc1c4d80b1f7486509550c4706f65e1882b1dcf7ec38a131 Nov 02 07:41:31.619875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:31.619832 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" event={"ID":"faa74a97-484e-417c-9a5a-fb45c815b342","Type":"ContainerStarted","Data":"0c2dd6078f7f31a98d12c420a3f0efd2ade7452f9bdde336b3321566f9bed3c6"} Nov 02 07:41:31.620369 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:31.619885 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" event={"ID":"faa74a97-484e-417c-9a5a-fb45c815b342","Type":"ContainerStarted","Data":"1baaa38efa9d5e50dc1c4d80b1f7486509550c4706f65e1882b1dcf7ec38a131"} Nov 02 07:41:31.646155 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:31.646119 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:31.667828 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:31.667751 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" podStartSLOduration=1.667728025 podStartE2EDuration="1.667728025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:31.665591663 +0000 UTC m=+991.963872487" watchObservedRunningTime="2026-11-02 07:41:31.667728025 +0000 UTC m=+991.966008857" Nov 02 07:41:32.550268 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:32.550230 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:33.486599 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:33.486565 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:35.049131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.049091 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml"] Nov 02 07:41:35.054429 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.054407 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.088978 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.088938 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml"] Nov 02 07:41:35.089229 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.089054 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fe085e-serving-cert\"" Nov 02 07:41:35.178476 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.178442 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:35.182073 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.182044 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99fc02b-fc76-4659-86ad-4700fe4398d4-openshift-service-ca\") pod \"maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.182243 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.182091 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2v88\" (UniqueName: \"kubernetes.io/projected/f99fc02b-fc76-4659-86ad-4700fe4398d4-kube-api-access-n2v88\") pod \"maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.182243 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.182175 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-maas-api-tls\") pod \"maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.182243 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.182197 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-metrics-certs\") pod \"maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.283101 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.283060 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99fc02b-fc76-4659-86ad-4700fe4398d4-openshift-service-ca\") pod \"maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.283339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.283121 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2v88\" (UniqueName: \"kubernetes.io/projected/f99fc02b-fc76-4659-86ad-4700fe4398d4-kube-api-access-n2v88\") pod \"maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.283339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.283200 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-maas-api-tls\") pod \"maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.283339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.283225 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-metrics-certs\") pod \"maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.283798 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.283772 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99fc02b-fc76-4659-86ad-4700fe4398d4-openshift-service-ca\") pod \"maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.286001 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.285980 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-maas-api-tls\") pod \"maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.286213 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.285986 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-metrics-certs\") pod \"maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.306237 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.306169 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2v88\" (UniqueName: \"kubernetes.io/projected/f99fc02b-fc76-4659-86ad-4700fe4398d4-kube-api-access-n2v88\") pod \"maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.321820 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.321774 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:35.368794 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.368760 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:35.457271 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.457130 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:35.678892 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:35.678866 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml"] Nov 02 07:41:35.680480 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:41:35.680455 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99fc02b_fc76_4659_86ad_4700fe4398d4.slice/crio-564d8d91c5fa187953ec041ecc61f3c56b34c9d02af9df2f818d9aa45841a08e WatchSource:0}: Error finding container 564d8d91c5fa187953ec041ecc61f3c56b34c9d02af9df2f818d9aa45841a08e: Status 404 returned error can't find the container with id 564d8d91c5fa187953ec041ecc61f3c56b34c9d02af9df2f818d9aa45841a08e Nov 02 07:41:36.627035 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:36.627006 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:41:36.657288 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:36.657254 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" event={"ID":"f99fc02b-fc76-4659-86ad-4700fe4398d4","Type":"ContainerStarted","Data":"b8e03a29bc20578176d2f14e72cbec0c9dabd3f08f9c3452c3646b9323ec9286"} Nov 02 07:41:36.657494 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:36.657311 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:36.657494 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:36.657335 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" event={"ID":"f99fc02b-fc76-4659-86ad-4700fe4398d4","Type":"ContainerStarted","Data":"564d8d91c5fa187953ec041ecc61f3c56b34c9d02af9df2f818d9aa45841a08e"} Nov 02 07:41:36.715433 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:36.715382 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" podStartSLOduration=1.393054268 podStartE2EDuration="1.715369591s" totalImagesPullingTime="322.315323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:36.713852222 +0000 UTC m=+997.012133049" watchObservedRunningTime="2026-11-02 07:41:36.715369591 +0000 UTC m=+997.013650415" Nov 02 07:41:37.044923 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:37.044890 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:38.739647 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:38.739615 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:40.491558 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:40.491523 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:40.550184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:40.550136 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:40.563228 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:40.563196 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:40.691929 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:40.691855 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:41:42.207939 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:42.207905 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:42.668610 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:42.668579 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:41:43.141135 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:41:43.141097 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:41:43.987052 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:43.987019 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:46.011268 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:46.011233 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:48.443593 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:48.443549 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:50.082869 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:50.082825 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:50.482640 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:50.482609 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:50.700954 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:50.700918 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:51.184471 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:51.184433 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:51.405882 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:51.405843 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:52.037723 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.037689 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp"] Nov 02 07:41:52.042318 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.042289 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.074770 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.074737 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp"] Nov 02 07:41:52.115292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.115232 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.115494 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.115383 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.115494 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.115451 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.115610 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.115539 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.115610 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.115573 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.115785 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.115628 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.115785 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.115666 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56xvv\" (UniqueName: \"kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-kube-api-access-56xvv\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.115785 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.115702 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.115945 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.115802 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.216926 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.216885 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.217413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.216954 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.217413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.216991 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.217413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.217006 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.217413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.217155 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.217413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.217259 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56xvv\" (UniqueName: \"kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-kube-api-access-56xvv\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.217413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.217317 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.217413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.217348 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.217752 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.217452 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.217752 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.217452 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.217752 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.217688 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.217752 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.217720 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.217883 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.217858 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.218267 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.218237 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.219630 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.219610 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.219961 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.219940 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.258759 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.258723 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56xvv\" (UniqueName: \"kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-kube-api-access-56xvv\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.258952 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.258732 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.362610 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.362521 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:52.683554 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.683521 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp"] Nov 02 07:41:52.685220 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:41:52.685192 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f72f7a8_6385_4190_9a7c_beecdd3ef7ef.slice/crio-0e88263f2c6c7128f93e42bc1b6cb1886d85b60b2ccbb11c11ec28ca9b41bdaf WatchSource:0}: Error finding container 0e88263f2c6c7128f93e42bc1b6cb1886d85b60b2ccbb11c11ec28ca9b41bdaf: Status 404 returned error can't find the container with id 0e88263f2c6c7128f93e42bc1b6cb1886d85b60b2ccbb11c11ec28ca9b41bdaf Nov 02 07:41:52.687465 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.687432 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:52.687805 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.687776 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:41:52.687890 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.687842 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:41:52.687890 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.687870 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:41:52.769242 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:52.769207 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" event={"ID":"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef","Type":"ContainerStarted","Data":"0e88263f2c6c7128f93e42bc1b6cb1886d85b60b2ccbb11c11ec28ca9b41bdaf"} Nov 02 07:41:53.794374 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:53.794332 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" event={"ID":"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef","Type":"ContainerStarted","Data":"ea4e86594e26b1767625bf95560905feea104ece72ad13d1d85d2857f7446b67"} Nov 02 07:41:53.883185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:53.856021 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" podStartSLOduration=1.856000828 podStartE2EDuration="1.856000828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:53.854812679 +0000 UTC m=+1014.153093505" watchObservedRunningTime="2026-11-02 07:41:53.856000828 +0000 UTC m=+1014.154281655" Nov 02 07:41:54.372187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:54.366481 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:41:54.372187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:54.370274 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 07:41:54.372187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:54.370332 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" podUID="6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 07:41:55.118412 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:55.118373 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:55.367186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:55.364608 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 07:41:55.367186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:55.364671 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" podUID="6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 07:41:56.183184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:56.180084 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:56.337150 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:56.337108 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:56.370604 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:56.370568 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 07:41:56.386300 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:56.386246 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" podUID="6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 07:41:57.344385 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:57.344343 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:57.365181 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:57.365134 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 07:41:57.365440 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:57.365410 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" podUID="6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 07:41:58.013628 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:58.013562 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:58.181517 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:58.181475 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:58.391242 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:58.390219 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 07:41:58.391242 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:58.390311 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" podUID="6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 07:41:59.362915 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:59.362877 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 07:41:59.363089 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:59.362936 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" podUID="6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 07:41:59.497070 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:59.497039 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:41:59.995583 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:41:59.995545 5899 scope.go:122] "RemoveContainer" containerID="6c9e3f4e8b16f1bf71005613b763cf47560e8745ade1da1c6c7d5441ed01838e" Nov 02 07:42:00.243243 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:00.243214 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:00.368293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:00.368217 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:42:00.368574 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:00.368547 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:42:00.369462 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:00.369440 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 07:42:00.957593 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:00.957560 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:01.462113 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:01.462064 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:01.652098 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:01.652052 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:02.087589 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:02.087555 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:02.278936 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:02.278900 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:03.817191 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:03.817145 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:04.789827 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:04.789796 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:04.933768 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:04.933733 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:05.916078 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:05.916049 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:06.806382 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:06.806348 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:06.948294 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:06.948259 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:08.465876 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:08.465844 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:10.397202 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:10.397146 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:12.247866 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:12.247830 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:13.143775 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:42:13.143737 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:42:13.945083 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:13.945047 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs"] Nov 02 07:42:13.950825 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:13.950805 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:13.979365 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:13.979334 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cc2bcd-serving-cert\"" Nov 02 07:42:13.983595 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:13.983566 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs"] Nov 02 07:42:14.079674 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.079642 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:14.111145 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.111108 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-metrics-certs\") pod \"maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:14.111329 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.111231 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6v6q\" (UniqueName: \"kubernetes.io/projected/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-kube-api-access-k6v6q\") pod \"maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:14.111329 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.111321 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-maas-api-tls\") pod \"maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:14.111438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.111424 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-openshift-service-ca\") pod \"maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:14.212944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.212865 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-maas-api-tls\") pod \"maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:14.213102 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.212952 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-openshift-service-ca\") pod \"maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:14.213102 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.213059 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-metrics-certs\") pod \"maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:14.213220 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.213108 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6v6q\" (UniqueName: \"kubernetes.io/projected/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-kube-api-access-k6v6q\") pod \"maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:14.213734 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.213706 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-openshift-service-ca\") pod \"maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:14.214510 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.214484 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:14.215576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.215548 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-maas-api-tls\") pod \"maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:14.215668 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.215581 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-metrics-certs\") pod \"maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:14.236801 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.236769 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6v6q\" (UniqueName: \"kubernetes.io/projected/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-kube-api-access-k6v6q\") pod \"maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:14.264619 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.264588 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:14.583624 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.583596 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs"] Nov 02 07:42:14.585246 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:42:14.585214 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e85108_b65c_43f1_8cc5_c435dd9e91b8.slice/crio-75760375d2ca830a0cebc1ba590430f067dbd0e5273c5a46079c24a2fd60f368 WatchSource:0}: Error finding container 75760375d2ca830a0cebc1ba590430f067dbd0e5273c5a46079c24a2fd60f368: Status 404 returned error can't find the container with id 75760375d2ca830a0cebc1ba590430f067dbd0e5273c5a46079c24a2fd60f368 Nov 02 07:42:14.929061 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:14.928968 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" event={"ID":"c3e85108-b65c-43f1-8cc5-c435dd9e91b8","Type":"ContainerStarted","Data":"75760375d2ca830a0cebc1ba590430f067dbd0e5273c5a46079c24a2fd60f368"} Nov 02 07:42:15.936677 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:15.936640 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" event={"ID":"c3e85108-b65c-43f1-8cc5-c435dd9e91b8","Type":"ContainerStarted","Data":"1bd7c33cf6654d863b7a730afd9e61b5992ac06c37581c7a78e906a37aabe864"} Nov 02 07:42:15.937086 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:15.936706 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:15.963776 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:15.963718 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:15.979626 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:15.979580 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" podStartSLOduration=2.543458742 podStartE2EDuration="2.979567078s" totalImagesPullingTime="436.108336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:42:15.978101488 +0000 UTC m=+1036.276382332" watchObservedRunningTime="2026-11-02 07:42:15.979567078 +0000 UTC m=+1036.277847953" Nov 02 07:42:17.704281 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:17.704239 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:19.507292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:19.507260 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:21.313513 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:21.313478 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:21.949705 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:21.949674 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:42:22.960206 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:22.960153 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r"] Nov 02 07:42:22.964686 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:22.964664 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:22.981882 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:22.981858 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b3e5f7-serving-cert\"" Nov 02 07:42:22.994230 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:22.994197 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r"] Nov 02 07:42:23.031773 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.031739 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/672e0db2-5a83-42b7-a15f-070354ce182b-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:23.031961 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.031801 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-metrics-certs\") pod \"maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:23.032045 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.032022 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8kb6\" (UniqueName: \"kubernetes.io/projected/672e0db2-5a83-42b7-a15f-070354ce182b-kube-api-access-z8kb6\") pod \"maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:23.032112 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.032087 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-maas-api-tls\") pod \"maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:23.086723 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.086678 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:23.133455 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.133417 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8kb6\" (UniqueName: \"kubernetes.io/projected/672e0db2-5a83-42b7-a15f-070354ce182b-kube-api-access-z8kb6\") pod \"maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:23.133610 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.133483 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-maas-api-tls\") pod \"maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:23.133610 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.133509 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/672e0db2-5a83-42b7-a15f-070354ce182b-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:23.133610 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.133560 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-metrics-certs\") pod \"maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:23.134194 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.134135 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/672e0db2-5a83-42b7-a15f-070354ce182b-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:23.136037 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.136011 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-maas-api-tls\") pod \"maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:23.136037 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.136022 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-metrics-certs\") pod \"maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:23.158593 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.158565 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8kb6\" (UniqueName: \"kubernetes.io/projected/672e0db2-5a83-42b7-a15f-070354ce182b-kube-api-access-z8kb6\") pod \"maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:23.217695 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.217611 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:23.279559 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.279525 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:23.365829 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.365680 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:23.594351 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.594325 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r"] Nov 02 07:42:23.595646 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:42:23.595614 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672e0db2_5a83_42b7_a15f_070354ce182b.slice/crio-794eb34260c2e8ed26562c601bf232358a81fc7bd509ffb0bd79cb2e855a50f9 WatchSource:0}: Error finding container 794eb34260c2e8ed26562c601bf232358a81fc7bd509ffb0bd79cb2e855a50f9: Status 404 returned error can't find the container with id 794eb34260c2e8ed26562c601bf232358a81fc7bd509ffb0bd79cb2e855a50f9 Nov 02 07:42:23.992625 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:23.992587 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" event={"ID":"672e0db2-5a83-42b7-a15f-070354ce182b","Type":"ContainerStarted","Data":"794eb34260c2e8ed26562c601bf232358a81fc7bd509ffb0bd79cb2e855a50f9"} Nov 02 07:42:25.001762 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:25.001713 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" event={"ID":"672e0db2-5a83-42b7-a15f-070354ce182b","Type":"ContainerStarted","Data":"99ab079ff271ba8c640a05440946b02716bf3a780ea09065d897c0bb1450be18"} Nov 02 07:42:25.002299 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:25.001786 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:25.012003 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:25.011974 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:25.044943 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:25.044887 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" podStartSLOduration=2.702396999 podStartE2EDuration="3.044868328s" totalImagesPullingTime="342.471329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:42:25.042522699 +0000 UTC m=+1045.340803544" watchObservedRunningTime="2026-11-02 07:42:25.044868328 +0000 UTC m=+1045.343149162" Nov 02 07:42:26.806700 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:26.806637 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:28.503232 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:28.503194 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:30.307906 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:30.307866 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:31.013741 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:31.013709 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:42:32.081981 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:32.081952 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:33.935540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:33.935503 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:36.342987 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:36.342944 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:36.725537 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:36.725441 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:38.229629 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:38.229593 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:38.372226 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:38.372195 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:38.515217 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:38.515137 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:39.267328 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:39.267293 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:40.409691 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:40.409654 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:42.305947 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:42.305912 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:43.140919 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:42:43.140886 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:42:44.071592 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:44.071559 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:44.690411 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:44.690379 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:44.867013 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:44.866974 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:46.492364 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:46.492334 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:46.629970 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:46.629928 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:47.545825 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:47.545764 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:47.683466 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:47.683431 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:48.530903 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:48.530871 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:50.367527 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:50.367490 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:50.509919 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:50.509884 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:51.122347 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:51.122313 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:52.405512 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:52.405474 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:52.551139 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:52.551102 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:52.684122 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:52.684040 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:53.239952 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:53.239906 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:54.064343 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:54.064308 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:55.932880 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:55.932848 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:57.929094 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:57.929057 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:58.177504 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:58.177470 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:58.540333 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:58.540290 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:58.683992 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:58.683932 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:42:59.322279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:42:59.322244 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:00.000500 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:00.000465 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:01.821364 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:01.821333 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:02.564186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:02.560628 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:04.117077 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:04.117022 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:05.908356 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:05.908323 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g"] Nov 02 07:43:05.908821 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:05.908529 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" podUID="b9077deb-a707-4161-b0bc-85899c58fca3" containerName="maas-api" containerID="cri-o://27ab57106c0103abb4e56453d02d5e101c34225371fe3ff8af2eeba1113df5f7" gracePeriod=30 Nov 02 07:43:06.038126 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.038093 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:06.189191 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.189095 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:06.242375 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.242018 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:43:06.301186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.291362 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9077deb-a707-4161-b0bc-85899c58fca3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9077deb-a707-4161-b0bc-85899c58fca3-openshift-service-ca\") pod \"b9077deb-a707-4161-b0bc-85899c58fca3\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " Nov 02 07:43:06.301186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.291617 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-metrics-certs\") pod \"b9077deb-a707-4161-b0bc-85899c58fca3\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " Nov 02 07:43:06.301186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.291697 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-maas-api-tls\") pod \"b9077deb-a707-4161-b0bc-85899c58fca3\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " Nov 02 07:43:06.301186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.291740 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9077deb-a707-4161-b0bc-85899c58fca3-kube-api-access-tctvj\" (UniqueName: \"kubernetes.io/projected/b9077deb-a707-4161-b0bc-85899c58fca3-kube-api-access-tctvj\") pod \"b9077deb-a707-4161-b0bc-85899c58fca3\" (UID: \"b9077deb-a707-4161-b0bc-85899c58fca3\") " Nov 02 07:43:06.309480 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.309444 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9077deb-a707-4161-b0bc-85899c58fca3-openshift-service-ca" pod "b9077deb-a707-4161-b0bc-85899c58fca3" (UID: "b9077deb-a707-4161-b0bc-85899c58fca3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:43:06.332720 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.332660 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9077deb-a707-4161-b0bc-85899c58fca3-kube-api-access-tctvj" pod "b9077deb-a707-4161-b0bc-85899c58fca3" (UID: "b9077deb-a707-4161-b0bc-85899c58fca3"). InnerVolumeSpecName "kube-api-access-tctvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:43:06.333009 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.332989 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-metrics-certs" pod "b9077deb-a707-4161-b0bc-85899c58fca3" (UID: "b9077deb-a707-4161-b0bc-85899c58fca3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:43:06.333644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.333621 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-maas-api-tls" pod "b9077deb-a707-4161-b0bc-85899c58fca3" (UID: "b9077deb-a707-4161-b0bc-85899c58fca3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:43:06.366193 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.362707 5899 generic.go:356] "Generic (PLEG): container finished" podID="b9077deb-a707-4161-b0bc-85899c58fca3" containerID="27ab57106c0103abb4e56453d02d5e101c34225371fe3ff8af2eeba1113df5f7" exitCode=0 Nov 02 07:43:06.366193 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.362744 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" event={"ID":"b9077deb-a707-4161-b0bc-85899c58fca3","Type":"ContainerDied","Data":"27ab57106c0103abb4e56453d02d5e101c34225371fe3ff8af2eeba1113df5f7"} Nov 02 07:43:06.366193 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.362809 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" event={"ID":"b9077deb-a707-4161-b0bc-85899c58fca3","Type":"ContainerDied","Data":"2d290e3df39a138520f26c3ebd972fbf25edfcbf620f748af72f5d77cccbd052"} Nov 02 07:43:06.366193 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.362829 5899 scope.go:122] "RemoveContainer" containerID="27ab57106c0103abb4e56453d02d5e101c34225371fe3ff8af2eeba1113df5f7" Nov 02 07:43:06.366193 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.363107 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g" Nov 02 07:43:06.371597 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.371569 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:06.397080 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.397043 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tctvj\" (UniqueName: \"kubernetes.io/projected/b9077deb-a707-4161-b0bc-85899c58fca3-kube-api-access-tctvj\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:06.397080 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.397083 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9077deb-a707-4161-b0bc-85899c58fca3-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:06.397305 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.397098 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:06.397305 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.397113 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9077deb-a707-4161-b0bc-85899c58fca3-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:06.397901 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.397870 5899 scope.go:122] "RemoveContainer" containerID="27ab57106c0103abb4e56453d02d5e101c34225371fe3ff8af2eeba1113df5f7" Nov 02 07:43:06.398335 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:43:06.398310 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ab57106c0103abb4e56453d02d5e101c34225371fe3ff8af2eeba1113df5f7\": container with ID starting with 27ab57106c0103abb4e56453d02d5e101c34225371fe3ff8af2eeba1113df5f7 not found: ID does not exist" containerID="27ab57106c0103abb4e56453d02d5e101c34225371fe3ff8af2eeba1113df5f7" Nov 02 07:43:06.398441 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.398345 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ab57106c0103abb4e56453d02d5e101c34225371fe3ff8af2eeba1113df5f7"} err="failed to get container status \"27ab57106c0103abb4e56453d02d5e101c34225371fe3ff8af2eeba1113df5f7\": rpc error: code = NotFound desc = could not find container \"27ab57106c0103abb4e56453d02d5e101c34225371fe3ff8af2eeba1113df5f7\": container with ID starting with 27ab57106c0103abb4e56453d02d5e101c34225371fe3ff8af2eeba1113df5f7 not found: ID does not exist" Nov 02 07:43:06.429050 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.429018 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g"] Nov 02 07:43:06.431614 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:06.431592 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d47676-8546fb879d-xzt8g"] Nov 02 07:43:07.148607 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.148573 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:07.276415 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.276378 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p"] Nov 02 07:43:07.277098 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.277074 5899 cpu_manager.go:414] "containerMap: removing container" podUID="b9077deb-a707-4161-b0bc-85899c58fca3" containerName="maas-api" Nov 02 07:43:07.277098 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.277097 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9077deb-a707-4161-b0bc-85899c58fca3" containerName="maas-api" Nov 02 07:43:07.277372 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.277355 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9077deb-a707-4161-b0bc-85899c58fca3" containerName="maas-api" Nov 02 07:43:07.282934 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.282915 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.320771 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.320738 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p"] Nov 02 07:43:07.369623 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.369583 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:07.416568 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.416485 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.416568 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.416538 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.416760 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.416689 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.416760 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.416712 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.416760 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.416733 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47s5v\" (UniqueName: \"kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-kube-api-access-47s5v\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.416870 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.416855 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.416906 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.416887 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.416985 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.416925 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.417065 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.417040 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.517834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.517798 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.517834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.517837 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.518062 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.517854 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47s5v\" (UniqueName: \"kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-kube-api-access-47s5v\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.518062 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.517904 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.518062 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.517930 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.518062 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.517967 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.518062 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.518017 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.518331 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.518106 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.518331 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.518210 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.518434 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.518370 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.518558 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.518538 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.518647 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.518576 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.518647 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.518612 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.518775 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.518755 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.520286 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.520267 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.520713 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.520688 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.541225 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.541200 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47s5v\" (UniqueName: \"kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-kube-api-access-47s5v\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.541353 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.541201 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.598588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.598549 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:07.820528 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.820498 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9077deb-a707-4161-b0bc-85899c58fca3" path="/var/lib/kubelet/pods/b9077deb-a707-4161-b0bc-85899c58fca3/volumes" Nov 02 07:43:07.919679 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.919649 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p"] Nov 02 07:43:07.921288 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:43:07.921262 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8490b8_4f35_42eb_9140_6e0bbc276f93.slice/crio-e7c512464f66a3cb9d72ff699acef99f4503cb1d510ab2af295822eba46cc574 WatchSource:0}: Error finding container e7c512464f66a3cb9d72ff699acef99f4503cb1d510ab2af295822eba46cc574: Status 404 returned error can't find the container with id e7c512464f66a3cb9d72ff699acef99f4503cb1d510ab2af295822eba46cc574 Nov 02 07:43:07.923778 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.923738 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:43:07.923877 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.923825 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:43:07.923877 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:07.923869 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:43:08.391323 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:08.391284 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" event={"ID":"bc8490b8-4f35-42eb-9140-6e0bbc276f93","Type":"ContainerStarted","Data":"49bb748d6f9dca01327695d67035c7df53813f0cddc8d7e5226899744cb38d5d"} Nov 02 07:43:08.391714 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:08.391348 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" event={"ID":"bc8490b8-4f35-42eb-9140-6e0bbc276f93","Type":"ContainerStarted","Data":"e7c512464f66a3cb9d72ff699acef99f4503cb1d510ab2af295822eba46cc574"} Nov 02 07:43:08.452959 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:08.452905 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" podStartSLOduration=1.452890131 podStartE2EDuration="1.452890131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:08.451703992 +0000 UTC m=+1088.749984842" watchObservedRunningTime="2026-11-02 07:43:08.452890131 +0000 UTC m=+1088.751170956" Nov 02 07:43:08.598797 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:08.598755 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:08.600246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:08.600215 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 07:43:08.600383 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:08.600270 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" podUID="bc8490b8-4f35-42eb-9140-6e0bbc276f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 07:43:09.095278 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:09.095245 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:09.319309 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:09.319270 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:09.474240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:09.473603 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:09.599829 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:09.599792 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 07:43:09.600002 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:09.599844 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" podUID="bc8490b8-4f35-42eb-9140-6e0bbc276f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 07:43:10.129523 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:10.129487 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:10.507944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:10.507537 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:10.603184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:10.600519 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 07:43:10.603184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:10.600581 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" podUID="bc8490b8-4f35-42eb-9140-6e0bbc276f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 07:43:11.339418 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:11.339386 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:11.499330 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:11.499301 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:11.599996 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:11.599909 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 07:43:11.600420 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:11.599996 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" podUID="bc8490b8-4f35-42eb-9140-6e0bbc276f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 07:43:12.598992 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:12.598941 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 07:43:12.599235 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:12.599009 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" podUID="bc8490b8-4f35-42eb-9140-6e0bbc276f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 07:43:12.983233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:12.980628 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml"] Nov 02 07:43:12.983233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:12.980882 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" podUID="f99fc02b-fc76-4659-86ad-4700fe4398d4" containerName="maas-api" containerID="cri-o://b8e03a29bc20578176d2f14e72cbec0c9dabd3f08f9c3452c3646b9323ec9286" gracePeriod=30 Nov 02 07:43:13.093494 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.093451 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:13.145414 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:43:13.145360 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:43:13.442048 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.442021 5899 generic.go:356] "Generic (PLEG): container finished" podID="f99fc02b-fc76-4659-86ad-4700fe4398d4" containerID="b8e03a29bc20578176d2f14e72cbec0c9dabd3f08f9c3452c3646b9323ec9286" exitCode=0 Nov 02 07:43:13.442288 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.442267 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" event={"ID":"f99fc02b-fc76-4659-86ad-4700fe4398d4","Type":"ContainerDied","Data":"b8e03a29bc20578176d2f14e72cbec0c9dabd3f08f9c3452c3646b9323ec9286"} Nov 02 07:43:13.442431 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.442419 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" event={"ID":"f99fc02b-fc76-4659-86ad-4700fe4398d4","Type":"ContainerDied","Data":"564d8d91c5fa187953ec041ecc61f3c56b34c9d02af9df2f818d9aa45841a08e"} Nov 02 07:43:13.442507 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.442498 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="564d8d91c5fa187953ec041ecc61f3c56b34c9d02af9df2f818d9aa45841a08e" Nov 02 07:43:13.447585 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.447562 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:43:13.599023 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.598991 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 07:43:13.599251 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.599056 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" podUID="bc8490b8-4f35-42eb-9140-6e0bbc276f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 07:43:13.624193 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.624129 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99fc02b-fc76-4659-86ad-4700fe4398d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99fc02b-fc76-4659-86ad-4700fe4398d4-openshift-service-ca\") pod \"f99fc02b-fc76-4659-86ad-4700fe4398d4\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " Nov 02 07:43:13.624193 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.624197 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-maas-api-tls\") pod \"f99fc02b-fc76-4659-86ad-4700fe4398d4\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " Nov 02 07:43:13.624410 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.624234 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-metrics-certs\") pod \"f99fc02b-fc76-4659-86ad-4700fe4398d4\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " Nov 02 07:43:13.624410 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.624386 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99fc02b-fc76-4659-86ad-4700fe4398d4-kube-api-access-n2v88\" (UniqueName: \"kubernetes.io/projected/f99fc02b-fc76-4659-86ad-4700fe4398d4-kube-api-access-n2v88\") pod \"f99fc02b-fc76-4659-86ad-4700fe4398d4\" (UID: \"f99fc02b-fc76-4659-86ad-4700fe4398d4\") " Nov 02 07:43:13.624566 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.624535 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99fc02b-fc76-4659-86ad-4700fe4398d4-openshift-service-ca" pod "f99fc02b-fc76-4659-86ad-4700fe4398d4" (UID: "f99fc02b-fc76-4659-86ad-4700fe4398d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:43:13.624896 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.624869 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99fc02b-fc76-4659-86ad-4700fe4398d4-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:13.626446 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.626423 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-maas-api-tls" pod "f99fc02b-fc76-4659-86ad-4700fe4398d4" (UID: "f99fc02b-fc76-4659-86ad-4700fe4398d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:43:13.626540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.626454 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-metrics-certs" pod "f99fc02b-fc76-4659-86ad-4700fe4398d4" (UID: "f99fc02b-fc76-4659-86ad-4700fe4398d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:43:13.626540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.626496 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99fc02b-fc76-4659-86ad-4700fe4398d4-kube-api-access-n2v88" pod "f99fc02b-fc76-4659-86ad-4700fe4398d4" (UID: "f99fc02b-fc76-4659-86ad-4700fe4398d4"). InnerVolumeSpecName "kube-api-access-n2v88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:43:13.730570 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.729777 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:13.730570 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.729807 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99fc02b-fc76-4659-86ad-4700fe4398d4-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:13.730570 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:13.729822 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2v88\" (UniqueName: \"kubernetes.io/projected/f99fc02b-fc76-4659-86ad-4700fe4398d4-kube-api-access-n2v88\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:14.260533 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:14.260481 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:14.448912 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:14.448876 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml" Nov 02 07:43:14.499996 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:14.499963 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml"] Nov 02 07:43:14.505710 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:14.505686 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe085e-8549c64bb9-tj7ml"] Nov 02 07:43:14.599407 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:14.599305 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 07:43:14.599407 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:14.599362 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" podUID="bc8490b8-4f35-42eb-9140-6e0bbc276f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 07:43:14.839256 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:14.839224 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:15.325594 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:15.325524 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:15.739753 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:15.739721 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:15.740205 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:15.740102 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:15.741305 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:15.741286 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 07:43:15.824205 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:15.821852 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99fc02b-fc76-4659-86ad-4700fe4398d4" path="/var/lib/kubelet/pods/f99fc02b-fc76-4659-86ad-4700fe4398d4/volumes" Nov 02 07:43:16.015249 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:16.015134 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:16.862234 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:16.862192 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:18.126468 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:18.126415 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:18.281420 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:18.281378 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:18.941245 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:18.941209 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:19.243387 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:19.243347 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:19.682026 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:19.681908 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:19.870075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:19.870027 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:21.447757 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:21.447692 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:21.640857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:21.640816 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:23.571225 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:23.571157 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:24.516423 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:24.516393 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:24.688397 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:24.688354 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:25.341790 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:25.341746 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:25.762051 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:25.762021 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:25.978476 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:25.978440 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:26.615095 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:26.615056 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:27.287246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:27.287211 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:28.143437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:28.143389 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:29.936955 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:29.936911 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:31.492788 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:31.492753 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs"] Nov 02 07:43:31.493367 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:31.492967 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" podUID="c3e85108-b65c-43f1-8cc5-c435dd9e91b8" containerName="maas-api" containerID="cri-o://1bd7c33cf6654d863b7a730afd9e61b5992ac06c37581c7a78e906a37aabe864" gracePeriod=30 Nov 02 07:43:31.646581 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:31.646547 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:31.800680 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:31.800644 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:43:31.948252 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:31.948213 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-openshift-service-ca\") pod \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " Nov 02 07:43:31.948438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:31.948284 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-metrics-certs\") pod \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " Nov 02 07:43:31.948438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:31.948345 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-kube-api-access-k6v6q\" (UniqueName: \"kubernetes.io/projected/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-kube-api-access-k6v6q\") pod \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " Nov 02 07:43:31.948566 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:31.948452 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-maas-api-tls\") pod \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\" (UID: \"c3e85108-b65c-43f1-8cc5-c435dd9e91b8\") " Nov 02 07:43:31.948624 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:31.948600 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-openshift-service-ca" pod "c3e85108-b65c-43f1-8cc5-c435dd9e91b8" (UID: "c3e85108-b65c-43f1-8cc5-c435dd9e91b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:43:31.948970 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:31.948951 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:31.952303 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:31.952272 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-metrics-certs" pod "c3e85108-b65c-43f1-8cc5-c435dd9e91b8" (UID: "c3e85108-b65c-43f1-8cc5-c435dd9e91b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:43:31.952432 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:31.952417 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-kube-api-access-k6v6q" pod "c3e85108-b65c-43f1-8cc5-c435dd9e91b8" (UID: "c3e85108-b65c-43f1-8cc5-c435dd9e91b8"). InnerVolumeSpecName "kube-api-access-k6v6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:43:31.954418 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:31.954395 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-maas-api-tls" pod "c3e85108-b65c-43f1-8cc5-c435dd9e91b8" (UID: "c3e85108-b65c-43f1-8cc5-c435dd9e91b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:43:32.049911 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:32.049826 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:32.049911 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:32.049857 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6v6q\" (UniqueName: \"kubernetes.io/projected/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-kube-api-access-k6v6q\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:32.049911 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:32.049869 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e85108-b65c-43f1-8cc5-c435dd9e91b8-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:32.111887 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:32.111846 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:32.602621 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:32.602588 5899 generic.go:356] "Generic (PLEG): container finished" podID="c3e85108-b65c-43f1-8cc5-c435dd9e91b8" containerID="1bd7c33cf6654d863b7a730afd9e61b5992ac06c37581c7a78e906a37aabe864" exitCode=0 Nov 02 07:43:32.603070 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:32.602637 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" event={"ID":"c3e85108-b65c-43f1-8cc5-c435dd9e91b8","Type":"ContainerDied","Data":"1bd7c33cf6654d863b7a730afd9e61b5992ac06c37581c7a78e906a37aabe864"} Nov 02 07:43:32.603070 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:32.602643 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" Nov 02 07:43:32.603070 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:32.602687 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs" event={"ID":"c3e85108-b65c-43f1-8cc5-c435dd9e91b8","Type":"ContainerDied","Data":"75760375d2ca830a0cebc1ba590430f067dbd0e5273c5a46079c24a2fd60f368"} Nov 02 07:43:32.603070 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:32.602711 5899 scope.go:122] "RemoveContainer" containerID="1bd7c33cf6654d863b7a730afd9e61b5992ac06c37581c7a78e906a37aabe864" Nov 02 07:43:32.613380 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:32.613360 5899 scope.go:122] "RemoveContainer" containerID="1bd7c33cf6654d863b7a730afd9e61b5992ac06c37581c7a78e906a37aabe864" Nov 02 07:43:32.613650 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:43:32.613629 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd7c33cf6654d863b7a730afd9e61b5992ac06c37581c7a78e906a37aabe864\": container with ID starting with 1bd7c33cf6654d863b7a730afd9e61b5992ac06c37581c7a78e906a37aabe864 not found: ID does not exist" containerID="1bd7c33cf6654d863b7a730afd9e61b5992ac06c37581c7a78e906a37aabe864" Nov 02 07:43:32.613708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:32.613658 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd7c33cf6654d863b7a730afd9e61b5992ac06c37581c7a78e906a37aabe864"} err="failed to get container status \"1bd7c33cf6654d863b7a730afd9e61b5992ac06c37581c7a78e906a37aabe864\": rpc error: code = NotFound desc = could not find container \"1bd7c33cf6654d863b7a730afd9e61b5992ac06c37581c7a78e906a37aabe864\": container with ID starting with 1bd7c33cf6654d863b7a730afd9e61b5992ac06c37581c7a78e906a37aabe864 not found: ID does not exist" Nov 02 07:43:32.683123 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:32.683090 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs"] Nov 02 07:43:32.694617 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:32.694571 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc2bcd-969d6b944-v72gs"] Nov 02 07:43:33.032812 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:33.032779 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:33.219436 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:33.219401 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:33.396430 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:33.396349 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:33.611097 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:33.611061 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:33.820318 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:33.820287 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e85108-b65c-43f1-8cc5-c435dd9e91b8" path="/var/lib/kubelet/pods/c3e85108-b65c-43f1-8cc5-c435dd9e91b8/volumes" Nov 02 07:43:34.803669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:34.803635 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:35.646790 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:35.646755 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:35.851872 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:35.851838 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:36.478996 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:36.478964 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:36.832631 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:36.832523 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:37.386454 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:37.386421 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:38.328818 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:38.327924 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:40.581422 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:40.581395 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:41.858465 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:41.858428 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:42.652044 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:42.652008 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:42.872153 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:42.872117 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r"] Nov 02 07:43:42.872640 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:42.872454 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" podUID="672e0db2-5a83-42b7-a15f-070354ce182b" containerName="maas-api" containerID="cri-o://99ab079ff271ba8c640a05440946b02716bf3a780ea09065d897c0bb1450be18" gracePeriod=30 Nov 02 07:43:43.038660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.038604 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:43.163211 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:43:43.162247 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:43:43.220182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.220064 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:43:43.347678 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.347591 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-metrics-certs\") pod \"672e0db2-5a83-42b7-a15f-070354ce182b\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " Nov 02 07:43:43.347863 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.347704 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672e0db2-5a83-42b7-a15f-070354ce182b-kube-api-access-z8kb6\" (UniqueName: \"kubernetes.io/projected/672e0db2-5a83-42b7-a15f-070354ce182b-kube-api-access-z8kb6\") pod \"672e0db2-5a83-42b7-a15f-070354ce182b\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " Nov 02 07:43:43.347863 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.347763 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-maas-api-tls\") pod \"672e0db2-5a83-42b7-a15f-070354ce182b\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " Nov 02 07:43:43.347979 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.347950 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/672e0db2-5a83-42b7-a15f-070354ce182b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/672e0db2-5a83-42b7-a15f-070354ce182b-openshift-service-ca\") pod \"672e0db2-5a83-42b7-a15f-070354ce182b\" (UID: \"672e0db2-5a83-42b7-a15f-070354ce182b\") " Nov 02 07:43:43.348403 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.348377 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672e0db2-5a83-42b7-a15f-070354ce182b-openshift-service-ca" pod "672e0db2-5a83-42b7-a15f-070354ce182b" (UID: "672e0db2-5a83-42b7-a15f-070354ce182b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:43:43.348582 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.348560 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/672e0db2-5a83-42b7-a15f-070354ce182b-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:43.349939 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.349913 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672e0db2-5a83-42b7-a15f-070354ce182b-kube-api-access-z8kb6" pod "672e0db2-5a83-42b7-a15f-070354ce182b" (UID: "672e0db2-5a83-42b7-a15f-070354ce182b"). InnerVolumeSpecName "kube-api-access-z8kb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:43:43.350063 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.350039 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-metrics-certs" pod "672e0db2-5a83-42b7-a15f-070354ce182b" (UID: "672e0db2-5a83-42b7-a15f-070354ce182b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:43:43.350279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.350259 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-maas-api-tls" pod "672e0db2-5a83-42b7-a15f-070354ce182b" (UID: "672e0db2-5a83-42b7-a15f-070354ce182b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:43:43.449943 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.449902 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:43.449943 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.449937 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8kb6\" (UniqueName: \"kubernetes.io/projected/672e0db2-5a83-42b7-a15f-070354ce182b-kube-api-access-z8kb6\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:43.450172 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.449954 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/672e0db2-5a83-42b7-a15f-070354ce182b-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:43:43.681378 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.681277 5899 generic.go:356] "Generic (PLEG): container finished" podID="672e0db2-5a83-42b7-a15f-070354ce182b" containerID="99ab079ff271ba8c640a05440946b02716bf3a780ea09065d897c0bb1450be18" exitCode=0 Nov 02 07:43:43.681556 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.681430 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" event={"ID":"672e0db2-5a83-42b7-a15f-070354ce182b","Type":"ContainerDied","Data":"99ab079ff271ba8c640a05440946b02716bf3a780ea09065d897c0bb1450be18"} Nov 02 07:43:43.681556 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.681479 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" event={"ID":"672e0db2-5a83-42b7-a15f-070354ce182b","Type":"ContainerDied","Data":"794eb34260c2e8ed26562c601bf232358a81fc7bd509ffb0bd79cb2e855a50f9"} Nov 02 07:43:43.681556 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.681508 5899 scope.go:122] "RemoveContainer" containerID="99ab079ff271ba8c640a05440946b02716bf3a780ea09065d897c0bb1450be18" Nov 02 07:43:43.681725 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.681617 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r" Nov 02 07:43:43.708548 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.708409 5899 scope.go:122] "RemoveContainer" containerID="99ab079ff271ba8c640a05440946b02716bf3a780ea09065d897c0bb1450be18" Nov 02 07:43:43.708875 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:43:43.708847 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ab079ff271ba8c640a05440946b02716bf3a780ea09065d897c0bb1450be18\": container with ID starting with 99ab079ff271ba8c640a05440946b02716bf3a780ea09065d897c0bb1450be18 not found: ID does not exist" containerID="99ab079ff271ba8c640a05440946b02716bf3a780ea09065d897c0bb1450be18" Nov 02 07:43:43.708982 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.708890 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ab079ff271ba8c640a05440946b02716bf3a780ea09065d897c0bb1450be18"} err="failed to get container status \"99ab079ff271ba8c640a05440946b02716bf3a780ea09065d897c0bb1450be18\": rpc error: code = NotFound desc = could not find container \"99ab079ff271ba8c640a05440946b02716bf3a780ea09065d897c0bb1450be18\": container with ID starting with 99ab079ff271ba8c640a05440946b02716bf3a780ea09065d897c0bb1450be18 not found: ID does not exist" Nov 02 07:43:43.764771 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.764731 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r"] Nov 02 07:43:43.774655 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.774627 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3e5f7-7cb4d48d6c-8hb7r"] Nov 02 07:43:43.820581 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:43.820545 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672e0db2-5a83-42b7-a15f-070354ce182b" path="/var/lib/kubelet/pods/672e0db2-5a83-42b7-a15f-070354ce182b/volumes" Nov 02 07:43:44.259058 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:44.259015 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:46.430765 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:46.430725 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:46.622656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:46.622621 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:46.974006 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:46.973970 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:47.572975 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:47.572931 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:48.711863 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:48.711832 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:50.232772 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.232733 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:50.418757 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.418716 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:50.626522 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.626446 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck"] Nov 02 07:43:50.627099 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.627065 5899 cpu_manager.go:414] "containerMap: removing container" podUID="672e0db2-5a83-42b7-a15f-070354ce182b" containerName="maas-api" Nov 02 07:43:50.627099 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.627082 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672e0db2-5a83-42b7-a15f-070354ce182b" containerName="maas-api" Nov 02 07:43:50.627219 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.627147 5899 cpu_manager.go:414] "containerMap: removing container" podUID="c3e85108-b65c-43f1-8cc5-c435dd9e91b8" containerName="maas-api" Nov 02 07:43:50.627219 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.627171 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e85108-b65c-43f1-8cc5-c435dd9e91b8" containerName="maas-api" Nov 02 07:43:50.627219 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.627188 5899 cpu_manager.go:414] "containerMap: removing container" podUID="f99fc02b-fc76-4659-86ad-4700fe4398d4" containerName="maas-api" Nov 02 07:43:50.627219 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.627193 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99fc02b-fc76-4659-86ad-4700fe4398d4" containerName="maas-api" Nov 02 07:43:50.627344 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.627303 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e85108-b65c-43f1-8cc5-c435dd9e91b8" containerName="maas-api" Nov 02 07:43:50.627344 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.627311 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99fc02b-fc76-4659-86ad-4700fe4398d4" containerName="maas-api" Nov 02 07:43:50.627344 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.627321 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="672e0db2-5a83-42b7-a15f-070354ce182b" containerName="maas-api" Nov 02 07:43:50.630922 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.630896 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:50.650377 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.650349 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-metrics-certs\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:50.650498 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.650403 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bea82b3-f00f-450e-947a-b134742e33d0-openshift-service-ca\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:50.650628 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.650612 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-maas-api-tls\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:50.650716 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.650701 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qr92\" (UniqueName: \"kubernetes.io/projected/5bea82b3-f00f-450e-947a-b134742e33d0-kube-api-access-2qr92\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:50.666114 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.666087 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-77dc65fb-serving-cert\"" Nov 02 07:43:50.680251 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.680222 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck"] Nov 02 07:43:50.683323 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.683295 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:50.752085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.752052 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-maas-api-tls\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:50.752291 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.752132 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qr92\" (UniqueName: \"kubernetes.io/projected/5bea82b3-f00f-450e-947a-b134742e33d0-kube-api-access-2qr92\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:50.752291 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:43:50.752239 5899 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-serving-cert: secret "maas-api-e2e-mt-77dc65fb-serving-cert" not found Nov 02 07:43:50.752291 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.752249 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-metrics-certs\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:50.752484 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:43:50.752303 5899 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-maas-api-tls podName:5bea82b3-f00f-450e-947a-b134742e33d0 nodeName:}" failed. No retries permitted until 2026-11-02 07:43:51.252283374 +0000 UTC m=+1131.550564193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-maas-api-tls") pod "maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" (UID: "5bea82b3-f00f-450e-947a-b134742e33d0") : secret "maas-api-e2e-mt-77dc65fb-serving-cert" not found Nov 02 07:43:50.752484 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.752321 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bea82b3-f00f-450e-947a-b134742e33d0-openshift-service-ca\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:50.753091 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.753068 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bea82b3-f00f-450e-947a-b134742e33d0-openshift-service-ca\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:50.754995 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.754964 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-metrics-certs\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:50.783191 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.783154 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qr92\" (UniqueName: \"kubernetes.io/projected/5bea82b3-f00f-450e-947a-b134742e33d0-kube-api-access-2qr92\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:50.945410 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:50.945327 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:51.268169 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:51.268124 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-maas-api-tls\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:51.270613 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:51.270587 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-maas-api-tls\") pod \"maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:51.385747 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:51.385709 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:51.547045 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:51.546957 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:51.807844 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:51.807765 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:51.866863 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:51.866832 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck"] Nov 02 07:43:51.868917 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:43:51.868884 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bea82b3_f00f_450e_947a_b134742e33d0.slice/crio-4914e40d8405cf758152363270c77a15dd92105ca8e6c396704d4337d0b443bd WatchSource:0}: Error finding container 4914e40d8405cf758152363270c77a15dd92105ca8e6c396704d4337d0b443bd: Status 404 returned error can't find the container with id 4914e40d8405cf758152363270c77a15dd92105ca8e6c396704d4337d0b443bd Nov 02 07:43:52.274969 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:52.274937 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:52.755751 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:52.755688 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" event={"ID":"5bea82b3-f00f-450e-947a-b134742e33d0","Type":"ContainerStarted","Data":"d1059b0305ad7cd49556f4f5c2c3bcfbc987031b56a2c8afaa506a6bc8f59991"} Nov 02 07:43:52.755934 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:52.755781 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" event={"ID":"5bea82b3-f00f-450e-947a-b134742e33d0","Type":"ContainerStarted","Data":"4914e40d8405cf758152363270c77a15dd92105ca8e6c396704d4337d0b443bd"} Nov 02 07:43:52.755996 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:52.755977 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:52.922965 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:52.922929 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:53.088709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:53.088623 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:53.962315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:53.962284 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:54.290779 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:54.290746 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:54.901209 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:54.901142 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:55.141399 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:55.141363 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:56.149902 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:56.149849 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:58.369660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:58.369605 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:58.768755 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:58.768727 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:43:58.821187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:58.821107 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" podStartSLOduration=8.49395136 podStartE2EDuration="8.821092205s" totalImagesPullingTime="327.140845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:52.808820198 +0000 UTC m=+1133.107101039" watchObservedRunningTime="2026-11-02 07:43:58.821092205 +0000 UTC m=+1139.119373031" Nov 02 07:43:58.845076 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:58.845033 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:43:59.005064 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:43:59.005024 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:00.475738 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:00.475651 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:00.862803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:00.862767 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:01.158506 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:01.158362 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:01.363884 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:01.363839 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:01.658008 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:01.657978 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:01.934010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:01.933934 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:02.183363 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:02.183329 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:02.356875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:02.356839 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:02.692476 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:02.692338 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:04.044462 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:04.044427 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:06.064762 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:06.064717 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:07.487669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:07.487634 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:07.850117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:07.850040 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:08.564717 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:08.564664 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:09.367794 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:09.367761 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:09.629111 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:09.629031 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:10.737956 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.737917 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs"] Nov 02 07:44:10.742317 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.742294 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:10.761113 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.761092 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-523de0-serving-cert\"" Nov 02 07:44:10.777777 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.777747 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs"] Nov 02 07:44:10.871867 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.871832 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-metrics-certs\") pod \"maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:10.872056 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.871974 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:10.872129 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.872046 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:10.872227 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.872206 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5fl6\" (UniqueName: \"kubernetes.io/projected/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-kube-api-access-t5fl6\") pod \"maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:10.883410 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.883364 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:10.973696 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.973667 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-metrics-certs\") pod \"maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:10.973869 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.973796 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:10.973869 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.973817 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:10.973942 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.973878 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5fl6\" (UniqueName: \"kubernetes.io/projected/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-kube-api-access-t5fl6\") pod \"maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:10.974541 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.974516 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:10.976351 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.976327 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:10.978989 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:10.978968 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-metrics-certs\") pod \"maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:11.002070 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:11.001992 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5fl6\" (UniqueName: \"kubernetes.io/projected/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-kube-api-access-t5fl6\") pod \"maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:11.061168 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:11.061113 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:11.286393 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:11.286279 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:11.391994 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:11.391969 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs"] Nov 02 07:44:11.394378 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:44:11.394354 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4c94a1_e9c1_4c9b_a85e_d12a8c3d0a1d.slice/crio-fac7d6636d782e94d4f224aa8ff5c6667308fc6a5bfb03258b98d48b185eb893 WatchSource:0}: Error finding container fac7d6636d782e94d4f224aa8ff5c6667308fc6a5bfb03258b98d48b185eb893: Status 404 returned error can't find the container with id fac7d6636d782e94d4f224aa8ff5c6667308fc6a5bfb03258b98d48b185eb893 Nov 02 07:44:11.891352 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:11.891308 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" event={"ID":"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d","Type":"ContainerStarted","Data":"4dae99236007216d9831e2798db3e8bad17c152826f1d1fc53673f4f42d663d7"} Nov 02 07:44:11.891793 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:11.891410 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" event={"ID":"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d","Type":"ContainerStarted","Data":"fac7d6636d782e94d4f224aa8ff5c6667308fc6a5bfb03258b98d48b185eb893"} Nov 02 07:44:11.891793 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:11.891438 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:11.981484 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:11.981432 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" podStartSLOduration=1.708428695 podStartE2EDuration="1.981415618s" totalImagesPullingTime="272.986923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:11.980720236 +0000 UTC m=+1152.279001060" watchObservedRunningTime="2026-11-02 07:44:11.981415618 +0000 UTC m=+1152.279696445" Nov 02 07:44:13.140861 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:44:13.140821 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:44:13.411233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:13.411116 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:14.266263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.266226 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx"] Nov 02 07:44:14.270698 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.270669 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.289707 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.289680 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 07:44:14.289842 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.289746 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 07:44:14.289842 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.289772 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bq7lr\"" Nov 02 07:44:14.301287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.301265 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 07:44:14.304814 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.304788 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx"] Nov 02 07:44:14.422899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.422868 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.423079 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.422920 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v4w8\" (UniqueName: \"kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-kube-api-access-8v4w8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.423079 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.422946 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.423177 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.423110 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.423177 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.423140 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.423287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.423262 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.423386 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.423368 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.524491 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.524399 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.524491 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.524464 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.524719 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.524563 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.524719 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.524656 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.524829 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.524715 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v4w8\" (UniqueName: \"kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-kube-api-access-8v4w8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.524829 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.524750 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.524926 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.524865 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.525274 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.525238 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.526913 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.526889 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.527010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.526974 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.527353 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.527330 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.527468 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.527449 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.547821 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.547795 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.548326 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.548307 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v4w8\" (UniqueName: \"kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-kube-api-access-8v4w8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ftdzx\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.584412 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.584385 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:14.906957 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:14.906922 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx"] Nov 02 07:44:14.908313 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:44:14.908288 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d2a632_5b70_4753_9ea0_090a7a41bbd8.slice/crio-98c165bf848222dea8c83072a99ede789f07b5a753fc9c645ba83a5880048db4 WatchSource:0}: Error finding container 98c165bf848222dea8c83072a99ede789f07b5a753fc9c645ba83a5880048db4: Status 404 returned error can't find the container with id 98c165bf848222dea8c83072a99ede789f07b5a753fc9c645ba83a5880048db4 Nov 02 07:44:15.840719 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:15.840676 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:15.922543 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:15.922505 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" event={"ID":"e7d2a632-5b70-4753-9ea0-090a7a41bbd8","Type":"ContainerStarted","Data":"98c165bf848222dea8c83072a99ede789f07b5a753fc9c645ba83a5880048db4"} Nov 02 07:44:17.442067 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:17.442031 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:44:17.442492 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:17.442094 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:44:17.903777 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:17.903746 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:44:17.937505 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:17.937466 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" event={"ID":"e7d2a632-5b70-4753-9ea0-090a7a41bbd8","Type":"ContainerStarted","Data":"b29f1bd22c274502bdbd3585e8143b3ef56e1f48995262102781a60b1ba58119"} Nov 02 07:44:17.937669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:17.937537 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:18.060052 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:18.060023 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:18.141075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:18.141005 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" podStartSLOduration=1.6107425869999998 podStartE2EDuration="4.140985696s" totalImagesPullingTime="2.530243109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:18.139346695 +0000 UTC m=+1158.437627522" watchObservedRunningTime="2026-11-02 07:44:18.140985696 +0000 UTC m=+1158.439266537" Nov 02 07:44:18.945823 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:18.945789 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:44:19.557748 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:19.557717 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:19.704971 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:19.704938 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:20.969452 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:20.969412 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:21.146469 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:21.146433 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:21.950518 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:21.950488 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:22.770370 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:22.770337 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:22.924694 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:22.924647 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:23.639451 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:23.639406 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:25.795668 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:25.795629 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:26.693900 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:26.693845 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:26.845834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:26.845798 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:28.003106 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:28.003069 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:28.494574 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:28.494520 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:28.650663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:28.650611 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:30.174931 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:30.174897 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:30.310561 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:30.310519 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:30.453644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:30.453546 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:31.669568 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:31.669529 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff"] Nov 02 07:44:31.857254 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:31.857216 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:32.133320 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.133283 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j"] Nov 02 07:44:32.137828 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.137806 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.153351 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.153323 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-523de0-openshift-default-dockercfg-ptktx\"" Nov 02 07:44:32.174314 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.174283 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j"] Nov 02 07:44:32.325311 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.325278 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-token\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.325489 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.325335 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a70f3e-6112-4665-b57a-bbe22502fec3-istiod-ca-cert\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.325554 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.325506 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-envoy\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.325554 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.325540 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-data\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.325672 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.325566 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-podinfo\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.325729 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.325709 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-credential-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.325785 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.325753 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-certs\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.325840 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.325807 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbgs9\" (UniqueName: \"kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-kube-api-access-pbgs9\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.325840 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.325837 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.426756 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.426665 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-token\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.426756 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.426731 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a70f3e-6112-4665-b57a-bbe22502fec3-istiod-ca-cert\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.426983 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.426916 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-envoy\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.426983 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.426962 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-data\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.427105 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.426996 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-podinfo\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.427105 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.427072 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-credential-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.427324 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.427115 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-certs\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.427324 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.427153 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbgs9\" (UniqueName: \"kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-kube-api-access-pbgs9\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.427324 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.427211 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.427494 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.427323 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-data\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.427606 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.427582 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.427873 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.427659 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-certs\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.427873 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.427671 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a70f3e-6112-4665-b57a-bbe22502fec3-istiod-ca-cert\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.427873 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.427677 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-credential-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.429201 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.429171 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-envoy\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.429484 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.429465 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-podinfo\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.451742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.451718 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-token\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.464806 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.464782 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbgs9\" (UniqueName: \"kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-kube-api-access-pbgs9\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:32.514994 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.514955 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:32.751410 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:32.751379 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:33.103427 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:33.103397 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j"] Nov 02 07:44:33.105938 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:44:33.105913 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a70f3e_6112_4665_b57a_bbe22502fec3.slice/crio-7726f4a856691a1ea39b55cb1acd2f08f8e26e2ef8eadd0e2ae5dd6d0bd035e1 WatchSource:0}: Error finding container 7726f4a856691a1ea39b55cb1acd2f08f8e26e2ef8eadd0e2ae5dd6d0bd035e1: Status 404 returned error can't find the container with id 7726f4a856691a1ea39b55cb1acd2f08f8e26e2ef8eadd0e2ae5dd6d0bd035e1 Nov 02 07:44:33.108547 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:33.108515 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:44:33.108637 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:33.108580 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:44:33.108637 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:33.108607 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:44:34.044673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:34.044636 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" event={"ID":"09a70f3e-6112-4665-b57a-bbe22502fec3","Type":"ContainerStarted","Data":"9877ab6cd4ce17203c0669a68ac0d3eb9f69a383069aee4eb312030b9d724dbb"} Nov 02 07:44:34.045279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:34.044694 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" event={"ID":"09a70f3e-6112-4665-b57a-bbe22502fec3","Type":"ContainerStarted","Data":"7726f4a856691a1ea39b55cb1acd2f08f8e26e2ef8eadd0e2ae5dd6d0bd035e1"} Nov 02 07:44:34.109358 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:34.109308 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" podStartSLOduration=2.109291365 podStartE2EDuration="2.109291365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:34.101799688 +0000 UTC m=+1174.400080513" watchObservedRunningTime="2026-11-02 07:44:34.109291365 +0000 UTC m=+1174.407572190" Nov 02 07:44:34.752337 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:34.752302 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:34.754189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:34.754140 5899 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 07:44:34.754309 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:34.754200 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" podUID="09a70f3e-6112-4665-b57a-bbe22502fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 07:44:34.812324 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:34.812250 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:35.751759 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:35.751729 5899 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 07:44:35.752213 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:35.751779 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" podUID="09a70f3e-6112-4665-b57a-bbe22502fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 07:44:36.381670 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:36.381635 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:36.752412 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:36.752379 5899 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 07:44:36.752790 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:36.752462 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" podUID="09a70f3e-6112-4665-b57a-bbe22502fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 07:44:36.999067 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:36.999031 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:37.751975 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:37.751946 5899 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 07:44:37.752269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:37.752244 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" podUID="09a70f3e-6112-4665-b57a-bbe22502fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 07:44:38.751744 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:38.751707 5899 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 07:44:38.752270 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:38.751764 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" podUID="09a70f3e-6112-4665-b57a-bbe22502fec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 07:44:39.419067 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:39.419027 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:39.758071 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:39.758039 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:39.758494 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:39.758475 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:39.759220 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:39.759200 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:44:41.582443 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:41.582245 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:41.820292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:41.820260 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:43.140943 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:44:43.140904 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:44:43.638791 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:43.638753 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:46.560566 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:46.560528 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:46.690725 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:46.690687 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" podUID="faa74a97-484e-417c-9a5a-fb45c815b342" containerName="main" containerID="cri-o://0c2dd6078f7f31a98d12c420a3f0efd2ade7452f9bdde336b3321566f9bed3c6" gracePeriod=45 Nov 02 07:44:46.979115 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:46.979082 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:44:47.066025 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.065987 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa74a97-484e-417c-9a5a-fb45c815b342-tls-certs\" (UniqueName: \"kubernetes.io/secret/faa74a97-484e-417c-9a5a-fb45c815b342-tls-certs\") pod \"faa74a97-484e-417c-9a5a-fb45c815b342\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " Nov 02 07:44:47.066223 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.066049 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-home\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-home\") pod \"faa74a97-484e-417c-9a5a-fb45c815b342\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " Nov 02 07:44:47.066223 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.066114 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-dshm\") pod \"faa74a97-484e-417c-9a5a-fb45c815b342\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " Nov 02 07:44:47.066223 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.066196 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-tmp-dir\") pod \"faa74a97-484e-417c-9a5a-fb45c815b342\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " Nov 02 07:44:47.066419 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.066287 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-model-cache\") pod \"faa74a97-484e-417c-9a5a-fb45c815b342\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " Nov 02 07:44:47.066419 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.066398 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa74a97-484e-417c-9a5a-fb45c815b342-kube-api-access-qbm5r\" (UniqueName: \"kubernetes.io/projected/faa74a97-484e-417c-9a5a-fb45c815b342-kube-api-access-qbm5r\") pod \"faa74a97-484e-417c-9a5a-fb45c815b342\" (UID: \"faa74a97-484e-417c-9a5a-fb45c815b342\") " Nov 02 07:44:47.066706 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.066664 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-tmp-dir" pod "faa74a97-484e-417c-9a5a-fb45c815b342" (UID: "faa74a97-484e-417c-9a5a-fb45c815b342"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:44:47.066706 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.066692 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-home" pod "faa74a97-484e-417c-9a5a-fb45c815b342" (UID: "faa74a97-484e-417c-9a5a-fb45c815b342"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:44:47.066917 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.066895 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-model-cache" pod "faa74a97-484e-417c-9a5a-fb45c815b342" (UID: "faa74a97-484e-417c-9a5a-fb45c815b342"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:44:47.067131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.067111 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:44:47.067238 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.067136 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:44:47.067238 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.067148 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:44:47.069351 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.069303 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-dshm" pod "faa74a97-484e-417c-9a5a-fb45c815b342" (UID: "faa74a97-484e-417c-9a5a-fb45c815b342"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:44:47.069454 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.069401 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa74a97-484e-417c-9a5a-fb45c815b342-tls-certs" pod "faa74a97-484e-417c-9a5a-fb45c815b342" (UID: "faa74a97-484e-417c-9a5a-fb45c815b342"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:44:47.069537 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.069518 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa74a97-484e-417c-9a5a-fb45c815b342-kube-api-access-qbm5r" pod "faa74a97-484e-417c-9a5a-fb45c815b342" (UID: "faa74a97-484e-417c-9a5a-fb45c815b342"). InnerVolumeSpecName "kube-api-access-qbm5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:44:47.169258 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.169148 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbm5r\" (UniqueName: \"kubernetes.io/projected/faa74a97-484e-417c-9a5a-fb45c815b342-kube-api-access-qbm5r\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:44:47.169258 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.169212 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa74a97-484e-417c-9a5a-fb45c815b342-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:44:47.169258 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.169228 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa74a97-484e-417c-9a5a-fb45c815b342-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:44:47.200743 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.200706 5899 generic.go:356] "Generic (PLEG): container finished" podID="faa74a97-484e-417c-9a5a-fb45c815b342" containerID="0c2dd6078f7f31a98d12c420a3f0efd2ade7452f9bdde336b3321566f9bed3c6" exitCode=0 Nov 02 07:44:47.200898 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.200756 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" Nov 02 07:44:47.200898 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.200800 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" event={"ID":"faa74a97-484e-417c-9a5a-fb45c815b342","Type":"ContainerDied","Data":"0c2dd6078f7f31a98d12c420a3f0efd2ade7452f9bdde336b3321566f9bed3c6"} Nov 02 07:44:47.200898 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.200845 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff" event={"ID":"faa74a97-484e-417c-9a5a-fb45c815b342","Type":"ContainerDied","Data":"1baaa38efa9d5e50dc1c4d80b1f7486509550c4706f65e1882b1dcf7ec38a131"} Nov 02 07:44:47.200898 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.200877 5899 scope.go:122] "RemoveContainer" containerID="0c2dd6078f7f31a98d12c420a3f0efd2ade7452f9bdde336b3321566f9bed3c6" Nov 02 07:44:47.211991 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.211972 5899 scope.go:122] "RemoveContainer" containerID="0c2dd6078f7f31a98d12c420a3f0efd2ade7452f9bdde336b3321566f9bed3c6" Nov 02 07:44:47.212291 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:44:47.212273 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2dd6078f7f31a98d12c420a3f0efd2ade7452f9bdde336b3321566f9bed3c6\": container with ID starting with 0c2dd6078f7f31a98d12c420a3f0efd2ade7452f9bdde336b3321566f9bed3c6 not found: ID does not exist" containerID="0c2dd6078f7f31a98d12c420a3f0efd2ade7452f9bdde336b3321566f9bed3c6" Nov 02 07:44:47.212370 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.212298 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2dd6078f7f31a98d12c420a3f0efd2ade7452f9bdde336b3321566f9bed3c6"} err="failed to get container status \"0c2dd6078f7f31a98d12c420a3f0efd2ade7452f9bdde336b3321566f9bed3c6\": rpc error: code = NotFound desc = could not find container \"0c2dd6078f7f31a98d12c420a3f0efd2ade7452f9bdde336b3321566f9bed3c6\": container with ID starting with 0c2dd6078f7f31a98d12c420a3f0efd2ade7452f9bdde336b3321566f9bed3c6 not found: ID does not exist" Nov 02 07:44:47.257558 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.257525 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff"] Nov 02 07:44:47.262083 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.262057 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-77dc65fb/e2e-lifecycle-model-77dc65fb-kserve-977d54d57-wvzff"] Nov 02 07:44:47.819894 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:47.819864 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa74a97-484e-417c-9a5a-fb45c815b342" path="/var/lib/kubelet/pods/faa74a97-484e-417c-9a5a-fb45c815b342/volumes" Nov 02 07:44:48.819794 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:48.819760 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:50.074363 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:50.074325 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:50.288710 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:50.288679 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:50.443918 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:50.443824 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:50.985565 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:50.985530 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:51.155538 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:51.155492 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:53.132413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:53.132380 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:55.029011 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:55.028949 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:56.811700 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:56.811664 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:59.144964 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:59.144926 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:44:59.508371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:44:59.508329 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:00.191683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.187762 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8"] Nov 02 07:45:00.191683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.188729 5899 cpu_manager.go:414] "containerMap: removing container" podUID="faa74a97-484e-417c-9a5a-fb45c815b342" containerName="main" Nov 02 07:45:00.191683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.188751 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa74a97-484e-417c-9a5a-fb45c815b342" containerName="main" Nov 02 07:45:00.191683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.188953 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa74a97-484e-417c-9a5a-fb45c815b342" containerName="main" Nov 02 07:45:00.194214 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.194190 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" Nov 02 07:45:00.217501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.217471 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd"] Nov 02 07:45:00.243174 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.243130 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" Nov 02 07:45:00.244236 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.243021 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8"] Nov 02 07:45:00.246040 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.245808 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd"] Nov 02 07:45:00.267978 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.267944 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e73a0c-facf-4baa-a3b7-48db4170f68f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8\" (UID: \"85e73a0c-facf-4baa-a3b7-48db4170f68f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" Nov 02 07:45:00.268492 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.268468 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rn2r\" (UniqueName: \"kubernetes.io/projected/85e73a0c-facf-4baa-a3b7-48db4170f68f-kube-api-access-8rn2r\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8\" (UID: \"85e73a0c-facf-4baa-a3b7-48db4170f68f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" Nov 02 07:45:00.281231 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.281197 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\"" Nov 02 07:45:00.369674 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.369591 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e73a0c-facf-4baa-a3b7-48db4170f68f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8\" (UID: \"85e73a0c-facf-4baa-a3b7-48db4170f68f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" Nov 02 07:45:00.369674 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.369664 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376db59c-582f-400a-a63c-45811479e258-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd\" (UID: \"376db59c-582f-400a-a63c-45811479e258\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" Nov 02 07:45:00.369899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.369719 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj5xh\" (UniqueName: \"kubernetes.io/projected/376db59c-582f-400a-a63c-45811479e258-kube-api-access-bj5xh\") pod \"maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd\" (UID: \"376db59c-582f-400a-a63c-45811479e258\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" Nov 02 07:45:00.370000 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.369978 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rn2r\" (UniqueName: \"kubernetes.io/projected/85e73a0c-facf-4baa-a3b7-48db4170f68f-kube-api-access-8rn2r\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8\" (UID: \"85e73a0c-facf-4baa-a3b7-48db4170f68f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" Nov 02 07:45:00.370320 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.370301 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e73a0c-facf-4baa-a3b7-48db4170f68f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8\" (UID: \"85e73a0c-facf-4baa-a3b7-48db4170f68f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" Nov 02 07:45:00.393047 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.393018 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rn2r\" (UniqueName: \"kubernetes.io/projected/85e73a0c-facf-4baa-a3b7-48db4170f68f-kube-api-access-8rn2r\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8\" (UID: \"85e73a0c-facf-4baa-a3b7-48db4170f68f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" Nov 02 07:45:00.471631 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.471594 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376db59c-582f-400a-a63c-45811479e258-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd\" (UID: \"376db59c-582f-400a-a63c-45811479e258\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" Nov 02 07:45:00.471820 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.471687 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5xh\" (UniqueName: \"kubernetes.io/projected/376db59c-582f-400a-a63c-45811479e258-kube-api-access-bj5xh\") pod \"maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd\" (UID: \"376db59c-582f-400a-a63c-45811479e258\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" Nov 02 07:45:00.472332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.472303 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376db59c-582f-400a-a63c-45811479e258-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd\" (UID: \"376db59c-582f-400a-a63c-45811479e258\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" Nov 02 07:45:00.494857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.494832 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5xh\" (UniqueName: \"kubernetes.io/projected/376db59c-582f-400a-a63c-45811479e258-kube-api-access-bj5xh\") pod \"maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd\" (UID: \"376db59c-582f-400a-a63c-45811479e258\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" Nov 02 07:45:00.516027 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.515994 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" Nov 02 07:45:00.568401 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.568361 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" Nov 02 07:45:00.817328 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.817298 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8"] Nov 02 07:45:00.818551 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:45:00.818522 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e73a0c_facf_4baa_a3b7_48db4170f68f.slice/crio-935879714c1af8b3e074d9799f8f22ba6de0c8f583779e3ec78d99505d88f487 WatchSource:0}: Error finding container 935879714c1af8b3e074d9799f8f22ba6de0c8f583779e3ec78d99505d88f487: Status 404 returned error can't find the container with id 935879714c1af8b3e074d9799f8f22ba6de0c8f583779e3ec78d99505d88f487 Nov 02 07:45:00.894538 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:00.894509 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd"] Nov 02 07:45:00.896273 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:45:00.896219 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376db59c_582f_400a_a63c_45811479e258.slice/crio-812b11896374ee2281a9cb531fe429aa157f0aeb19027b690c20bdd11e5a1ab7 WatchSource:0}: Error finding container 812b11896374ee2281a9cb531fe429aa157f0aeb19027b690c20bdd11e5a1ab7: Status 404 returned error can't find the container with id 812b11896374ee2281a9cb531fe429aa157f0aeb19027b690c20bdd11e5a1ab7 Nov 02 07:45:01.303433 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:01.303398 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:01.363716 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:01.363660 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" event={"ID":"376db59c-582f-400a-a63c-45811479e258","Type":"ContainerStarted","Data":"b7e4122e2b528da648f0d62ceb0e10d06045f990d556d9658af6013a5aea0930"} Nov 02 07:45:01.363716 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:01.363720 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" event={"ID":"376db59c-582f-400a-a63c-45811479e258","Type":"ContainerStarted","Data":"812b11896374ee2281a9cb531fe429aa157f0aeb19027b690c20bdd11e5a1ab7"} Nov 02 07:45:01.365550 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:01.365517 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" event={"ID":"85e73a0c-facf-4baa-a3b7-48db4170f68f","Type":"ContainerStarted","Data":"d30ace0355a8a84ddfd5a9cbbd59d2dc55dd88976bbc6a968e483c6cf4b70952"} Nov 02 07:45:01.365716 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:01.365569 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" event={"ID":"85e73a0c-facf-4baa-a3b7-48db4170f68f","Type":"ContainerStarted","Data":"935879714c1af8b3e074d9799f8f22ba6de0c8f583779e3ec78d99505d88f487"} Nov 02 07:45:01.406147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:01.406097 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" podStartSLOduration=1.406083048 podStartE2EDuration="1.406083048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:01.405108183 +0000 UTC m=+1201.703389008" watchObservedRunningTime="2026-11-02 07:45:01.406083048 +0000 UTC m=+1201.704363913" Nov 02 07:45:01.449957 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:01.449825 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" podStartSLOduration=1.44980463 podStartE2EDuration="1.44980463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:01.445278204 +0000 UTC m=+1201.743559029" watchObservedRunningTime="2026-11-02 07:45:01.44980463 +0000 UTC m=+1201.748085435" Nov 02 07:45:02.374752 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:02.374717 5899 generic.go:356] "Generic (PLEG): container finished" podID="376db59c-582f-400a-a63c-45811479e258" containerID="b7e4122e2b528da648f0d62ceb0e10d06045f990d556d9658af6013a5aea0930" exitCode=60 Nov 02 07:45:02.375243 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:02.374803 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" event={"ID":"376db59c-582f-400a-a63c-45811479e258","Type":"ContainerDied","Data":"b7e4122e2b528da648f0d62ceb0e10d06045f990d556d9658af6013a5aea0930"} Nov 02 07:45:02.375243 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:02.375106 5899 scope.go:122] "RemoveContainer" containerID="b7e4122e2b528da648f0d62ceb0e10d06045f990d556d9658af6013a5aea0930" Nov 02 07:45:02.376559 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:02.376539 5899 generic.go:356] "Generic (PLEG): container finished" podID="85e73a0c-facf-4baa-a3b7-48db4170f68f" containerID="d30ace0355a8a84ddfd5a9cbbd59d2dc55dd88976bbc6a968e483c6cf4b70952" exitCode=60 Nov 02 07:45:02.376658 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:02.376559 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" event={"ID":"85e73a0c-facf-4baa-a3b7-48db4170f68f","Type":"ContainerDied","Data":"d30ace0355a8a84ddfd5a9cbbd59d2dc55dd88976bbc6a968e483c6cf4b70952"} Nov 02 07:45:02.376834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:02.376821 5899 scope.go:122] "RemoveContainer" containerID="d30ace0355a8a84ddfd5a9cbbd59d2dc55dd88976bbc6a968e483c6cf4b70952" Nov 02 07:45:02.797032 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:02.796998 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:03.387643 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:03.387605 5899 generic.go:356] "Generic (PLEG): container finished" podID="376db59c-582f-400a-a63c-45811479e258" containerID="c9193ae1907f8f27de08001cea7fa140744b7b223f0bc28a7fd1b429589c1bb8" exitCode=60 Nov 02 07:45:03.388102 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:03.387661 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" event={"ID":"376db59c-582f-400a-a63c-45811479e258","Type":"ContainerDied","Data":"c9193ae1907f8f27de08001cea7fa140744b7b223f0bc28a7fd1b429589c1bb8"} Nov 02 07:45:03.388102 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:03.387803 5899 scope.go:122] "RemoveContainer" containerID="b7e4122e2b528da648f0d62ceb0e10d06045f990d556d9658af6013a5aea0930" Nov 02 07:45:03.388102 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:03.388061 5899 scope.go:122] "RemoveContainer" containerID="c9193ae1907f8f27de08001cea7fa140744b7b223f0bc28a7fd1b429589c1bb8" Nov 02 07:45:03.388470 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:45:03.388450 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd_odh-ai-gateway-infra(376db59c-582f-400a-a63c-45811479e258)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" podUID="376db59c-582f-400a-a63c-45811479e258" Nov 02 07:45:03.391088 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:03.391062 5899 generic.go:356] "Generic (PLEG): container finished" podID="85e73a0c-facf-4baa-a3b7-48db4170f68f" containerID="2faa0cd4ca28b466bff831d8b2b53e3a04e7d370ed6d822ba83258682326b098" exitCode=60 Nov 02 07:45:03.391315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:03.391294 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" event={"ID":"85e73a0c-facf-4baa-a3b7-48db4170f68f","Type":"ContainerDied","Data":"2faa0cd4ca28b466bff831d8b2b53e3a04e7d370ed6d822ba83258682326b098"} Nov 02 07:45:03.391449 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:03.391434 5899 scope.go:122] "RemoveContainer" containerID="2faa0cd4ca28b466bff831d8b2b53e3a04e7d370ed6d822ba83258682326b098" Nov 02 07:45:03.391729 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:45:03.391712 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8_odh-ai-gateway-infra(85e73a0c-facf-4baa-a3b7-48db4170f68f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" podUID="85e73a0c-facf-4baa-a3b7-48db4170f68f" Nov 02 07:45:03.403602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:03.403582 5899 scope.go:122] "RemoveContainer" containerID="d30ace0355a8a84ddfd5a9cbbd59d2dc55dd88976bbc6a968e483c6cf4b70952" Nov 02 07:45:03.424603 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:03.424547 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:03.591228 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:03.591156 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:04.401861 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:04.401829 5899 scope.go:122] "RemoveContainer" containerID="c9193ae1907f8f27de08001cea7fa140744b7b223f0bc28a7fd1b429589c1bb8" Nov 02 07:45:04.402298 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:45:04.402143 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd_odh-ai-gateway-infra(376db59c-582f-400a-a63c-45811479e258)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" podUID="376db59c-582f-400a-a63c-45811479e258" Nov 02 07:45:04.403567 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:04.403546 5899 scope.go:122] "RemoveContainer" containerID="2faa0cd4ca28b466bff831d8b2b53e3a04e7d370ed6d822ba83258682326b098" Nov 02 07:45:04.403823 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:45:04.403805 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8_odh-ai-gateway-infra(85e73a0c-facf-4baa-a3b7-48db4170f68f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" podUID="85e73a0c-facf-4baa-a3b7-48db4170f68f" Nov 02 07:45:05.061470 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:05.061436 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:05.228356 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:05.228322 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:06.620938 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:06.620899 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:08.145558 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:08.145528 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:08.420400 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:08.420313 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:08.722017 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:08.721933 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:09.333146 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:09.333113 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:10.417742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:10.417705 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:10.974558 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:10.974521 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:13.141606 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:45:13.141565 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:45:13.677292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:13.677256 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:15.009541 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:15.009509 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:15.185063 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:15.185024 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:16.556489 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:16.556449 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:16.715714 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:16.715669 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:16.812757 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:16.812685 5899 scope.go:122] "RemoveContainer" containerID="c9193ae1907f8f27de08001cea7fa140744b7b223f0bc28a7fd1b429589c1bb8" Nov 02 07:45:17.500889 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:17.500852 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" event={"ID":"376db59c-582f-400a-a63c-45811479e258","Type":"ContainerStarted","Data":"23ca90d38c4017564bd6a6de7e9dfb32d09c6ec96db8223127dd2636f8aea708"} Nov 02 07:45:17.812895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:17.812818 5899 scope.go:122] "RemoveContainer" containerID="2faa0cd4ca28b466bff831d8b2b53e3a04e7d370ed6d822ba83258682326b098" Nov 02 07:45:18.404938 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:18.404841 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:18.511033 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:18.511000 5899 generic.go:356] "Generic (PLEG): container finished" podID="376db59c-582f-400a-a63c-45811479e258" containerID="23ca90d38c4017564bd6a6de7e9dfb32d09c6ec96db8223127dd2636f8aea708" exitCode=60 Nov 02 07:45:18.511288 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:18.511076 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" event={"ID":"376db59c-582f-400a-a63c-45811479e258","Type":"ContainerDied","Data":"23ca90d38c4017564bd6a6de7e9dfb32d09c6ec96db8223127dd2636f8aea708"} Nov 02 07:45:18.511288 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:18.511204 5899 scope.go:122] "RemoveContainer" containerID="c9193ae1907f8f27de08001cea7fa140744b7b223f0bc28a7fd1b429589c1bb8" Nov 02 07:45:18.511703 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:18.511679 5899 scope.go:122] "RemoveContainer" containerID="23ca90d38c4017564bd6a6de7e9dfb32d09c6ec96db8223127dd2636f8aea708" Nov 02 07:45:18.512055 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:45:18.512032 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd_odh-ai-gateway-infra(376db59c-582f-400a-a63c-45811479e258)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" podUID="376db59c-582f-400a-a63c-45811479e258" Nov 02 07:45:18.514187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:18.514149 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" event={"ID":"85e73a0c-facf-4baa-a3b7-48db4170f68f","Type":"ContainerStarted","Data":"c9d0b6397c787b45c2d84b063a5b841c1439a0a6d1f4bbd7f1b57fe1b1cc4dfb"} Nov 02 07:45:18.552453 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:18.552418 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd"] Nov 02 07:45:19.524591 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:19.524550 5899 generic.go:356] "Generic (PLEG): container finished" podID="85e73a0c-facf-4baa-a3b7-48db4170f68f" containerID="c9d0b6397c787b45c2d84b063a5b841c1439a0a6d1f4bbd7f1b57fe1b1cc4dfb" exitCode=60 Nov 02 07:45:19.524591 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:19.524583 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" event={"ID":"85e73a0c-facf-4baa-a3b7-48db4170f68f","Type":"ContainerDied","Data":"c9d0b6397c787b45c2d84b063a5b841c1439a0a6d1f4bbd7f1b57fe1b1cc4dfb"} Nov 02 07:45:19.525138 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:19.524642 5899 scope.go:122] "RemoveContainer" containerID="2faa0cd4ca28b466bff831d8b2b53e3a04e7d370ed6d822ba83258682326b098" Nov 02 07:45:19.525138 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:19.525012 5899 scope.go:122] "RemoveContainer" containerID="c9d0b6397c787b45c2d84b063a5b841c1439a0a6d1f4bbd7f1b57fe1b1cc4dfb" Nov 02 07:45:19.525415 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:45:19.525387 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8_odh-ai-gateway-infra(85e73a0c-facf-4baa-a3b7-48db4170f68f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" podUID="85e73a0c-facf-4baa-a3b7-48db4170f68f" Nov 02 07:45:19.615300 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:19.615268 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8"] Nov 02 07:45:19.707239 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:19.707207 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" Nov 02 07:45:19.756980 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:19.756948 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/376db59c-582f-400a-a63c-45811479e258-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376db59c-582f-400a-a63c-45811479e258-maas-api-service-ca\") pod \"376db59c-582f-400a-a63c-45811479e258\" (UID: \"376db59c-582f-400a-a63c-45811479e258\") " Nov 02 07:45:19.757271 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:19.757252 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376db59c-582f-400a-a63c-45811479e258-kube-api-access-bj5xh\" (UniqueName: \"kubernetes.io/projected/376db59c-582f-400a-a63c-45811479e258-kube-api-access-bj5xh\") pod \"376db59c-582f-400a-a63c-45811479e258\" (UID: \"376db59c-582f-400a-a63c-45811479e258\") " Nov 02 07:45:19.757596 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:19.757571 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376db59c-582f-400a-a63c-45811479e258-maas-api-service-ca" pod "376db59c-582f-400a-a63c-45811479e258" (UID: "376db59c-582f-400a-a63c-45811479e258"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:45:19.757793 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:19.757777 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376db59c-582f-400a-a63c-45811479e258-maas-api-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:45:19.760056 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:19.760017 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376db59c-582f-400a-a63c-45811479e258-kube-api-access-bj5xh" pod "376db59c-582f-400a-a63c-45811479e258" (UID: "376db59c-582f-400a-a63c-45811479e258"). InnerVolumeSpecName "kube-api-access-bj5xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:45:19.859816 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:19.859780 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj5xh\" (UniqueName: \"kubernetes.io/projected/376db59c-582f-400a-a63c-45811479e258-kube-api-access-bj5xh\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:45:20.414419 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:20.414387 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:20.533492 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:20.533460 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" Nov 02 07:45:20.534726 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:20.533491 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd" event={"ID":"376db59c-582f-400a-a63c-45811479e258","Type":"ContainerDied","Data":"812b11896374ee2281a9cb531fe429aa157f0aeb19027b690c20bdd11e5a1ab7"} Nov 02 07:45:20.534726 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:20.533558 5899 scope.go:122] "RemoveContainer" containerID="23ca90d38c4017564bd6a6de7e9dfb32d09c6ec96db8223127dd2636f8aea708" Nov 02 07:45:20.635221 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:20.635190 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd"] Nov 02 07:45:20.641408 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:20.641375 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5a032f-29893425-ffkmd"] Nov 02 07:45:20.705094 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:20.705068 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" Nov 02 07:45:20.794540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:20.794507 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e73a0c-facf-4baa-a3b7-48db4170f68f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e73a0c-facf-4baa-a3b7-48db4170f68f-maas-api-service-ca\") pod \"85e73a0c-facf-4baa-a3b7-48db4170f68f\" (UID: \"85e73a0c-facf-4baa-a3b7-48db4170f68f\") " Nov 02 07:45:20.794836 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:20.794560 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e73a0c-facf-4baa-a3b7-48db4170f68f-kube-api-access-8rn2r\" (UniqueName: \"kubernetes.io/projected/85e73a0c-facf-4baa-a3b7-48db4170f68f-kube-api-access-8rn2r\") pod \"85e73a0c-facf-4baa-a3b7-48db4170f68f\" (UID: \"85e73a0c-facf-4baa-a3b7-48db4170f68f\") " Nov 02 07:45:20.794982 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:20.794957 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e73a0c-facf-4baa-a3b7-48db4170f68f-maas-api-service-ca" pod "85e73a0c-facf-4baa-a3b7-48db4170f68f" (UID: "85e73a0c-facf-4baa-a3b7-48db4170f68f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:45:20.795407 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:20.795383 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e73a0c-facf-4baa-a3b7-48db4170f68f-maas-api-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:45:20.797073 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:20.797051 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e73a0c-facf-4baa-a3b7-48db4170f68f-kube-api-access-8rn2r" pod "85e73a0c-facf-4baa-a3b7-48db4170f68f" (UID: "85e73a0c-facf-4baa-a3b7-48db4170f68f"). InnerVolumeSpecName "kube-api-access-8rn2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:45:20.896642 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:20.896606 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rn2r\" (UniqueName: \"kubernetes.io/projected/85e73a0c-facf-4baa-a3b7-48db4170f68f-kube-api-access-8rn2r\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:45:21.545627 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:21.545597 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" Nov 02 07:45:21.546035 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:21.545603 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8" event={"ID":"85e73a0c-facf-4baa-a3b7-48db4170f68f","Type":"ContainerDied","Data":"935879714c1af8b3e074d9799f8f22ba6de0c8f583779e3ec78d99505d88f487"} Nov 02 07:45:21.546035 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:21.545722 5899 scope.go:122] "RemoveContainer" containerID="c9d0b6397c787b45c2d84b063a5b841c1439a0a6d1f4bbd7f1b57fe1b1cc4dfb" Nov 02 07:45:21.598097 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:21.598056 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8"] Nov 02 07:45:21.609499 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:21.609464 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893425-2rxh8"] Nov 02 07:45:21.833227 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:21.833124 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="376db59c-582f-400a-a63c-45811479e258" path="/var/lib/kubelet/pods/376db59c-582f-400a-a63c-45811479e258/volumes" Nov 02 07:45:21.835185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:21.834430 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e73a0c-facf-4baa-a3b7-48db4170f68f" path="/var/lib/kubelet/pods/85e73a0c-facf-4baa-a3b7-48db4170f68f/volumes" Nov 02 07:45:22.389351 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:22.389315 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:22.634695 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:22.634650 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:23.327099 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:23.327063 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:23.843439 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:23.843404 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:23.986811 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:23.986696 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:24.460602 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:24.460568 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:24.924928 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:24.924898 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:25.056641 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:25.056610 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:26.427280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:26.427247 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:26.646462 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:26.646422 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:27.940439 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:27.940405 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:28.528441 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:28.528408 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:28.667560 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:28.667523 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:28.815135 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:28.815052 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:28.979217 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:28.979152 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:30.102999 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:30.102970 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:30.332493 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:30.332455 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:30.464623 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:30.464537 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:32.068951 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:32.068919 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:32.298192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:32.298137 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:32.503782 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:32.503720 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:33.028700 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:33.028667 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:33.207930 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:33.207887 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:34.546339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.546301 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck"] Nov 02 07:45:34.546827 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.546570 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" podUID="5bea82b3-f00f-450e-947a-b134742e33d0" containerName="maas-api" containerID="cri-o://d1059b0305ad7cd49556f4f5c2c3bcfbc987031b56a2c8afaa506a6bc8f59991" gracePeriod=30 Nov 02 07:45:34.688024 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.687983 5899 generic.go:356] "Generic (PLEG): container finished" podID="5bea82b3-f00f-450e-947a-b134742e33d0" containerID="d1059b0305ad7cd49556f4f5c2c3bcfbc987031b56a2c8afaa506a6bc8f59991" exitCode=0 Nov 02 07:45:34.688237 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.688053 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" event={"ID":"5bea82b3-f00f-450e-947a-b134742e33d0","Type":"ContainerDied","Data":"d1059b0305ad7cd49556f4f5c2c3bcfbc987031b56a2c8afaa506a6bc8f59991"} Nov 02 07:45:34.751018 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.750962 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:34.927181 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.927133 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:45:34.977505 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.977457 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bea82b3-f00f-450e-947a-b134742e33d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bea82b3-f00f-450e-947a-b134742e33d0-openshift-service-ca\") pod \"5bea82b3-f00f-450e-947a-b134742e33d0\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " Nov 02 07:45:34.977708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.977683 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bea82b3-f00f-450e-947a-b134742e33d0-kube-api-access-2qr92\" (UniqueName: \"kubernetes.io/projected/5bea82b3-f00f-450e-947a-b134742e33d0-kube-api-access-2qr92\") pod \"5bea82b3-f00f-450e-947a-b134742e33d0\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " Nov 02 07:45:34.977796 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.977747 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-metrics-certs\") pod \"5bea82b3-f00f-450e-947a-b134742e33d0\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " Nov 02 07:45:34.977875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.977854 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bea82b3-f00f-450e-947a-b134742e33d0-openshift-service-ca" pod "5bea82b3-f00f-450e-947a-b134742e33d0" (UID: "5bea82b3-f00f-450e-947a-b134742e33d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:45:34.977931 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.977917 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-maas-api-tls\") pod \"5bea82b3-f00f-450e-947a-b134742e33d0\" (UID: \"5bea82b3-f00f-450e-947a-b134742e33d0\") " Nov 02 07:45:34.978604 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.978555 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bea82b3-f00f-450e-947a-b134742e33d0-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:45:34.980369 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.980334 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bea82b3-f00f-450e-947a-b134742e33d0-kube-api-access-2qr92" pod "5bea82b3-f00f-450e-947a-b134742e33d0" (UID: "5bea82b3-f00f-450e-947a-b134742e33d0"). InnerVolumeSpecName "kube-api-access-2qr92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:45:34.981692 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.981664 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-maas-api-tls" pod "5bea82b3-f00f-450e-947a-b134742e33d0" (UID: "5bea82b3-f00f-450e-947a-b134742e33d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:45:34.982093 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:34.982065 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-metrics-certs" pod "5bea82b3-f00f-450e-947a-b134742e33d0" (UID: "5bea82b3-f00f-450e-947a-b134742e33d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:45:35.079959 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:35.079920 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qr92\" (UniqueName: \"kubernetes.io/projected/5bea82b3-f00f-450e-947a-b134742e33d0-kube-api-access-2qr92\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:45:35.079959 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:35.079952 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:45:35.079959 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:35.079961 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bea82b3-f00f-450e-947a-b134742e33d0-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:45:35.650479 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:35.650429 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:35.696395 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:35.696365 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" Nov 02 07:45:35.696395 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:35.696385 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck" event={"ID":"5bea82b3-f00f-450e-947a-b134742e33d0","Type":"ContainerDied","Data":"4914e40d8405cf758152363270c77a15dd92105ca8e6c396704d4337d0b443bd"} Nov 02 07:45:35.696641 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:35.696453 5899 scope.go:122] "RemoveContainer" containerID="d1059b0305ad7cd49556f4f5c2c3bcfbc987031b56a2c8afaa506a6bc8f59991" Nov 02 07:45:35.749309 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:35.749273 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck"] Nov 02 07:45:35.760781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:35.760753 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77dc65fb-8c79bd6f-s9fck"] Nov 02 07:45:35.820232 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:35.820201 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bea82b3-f00f-450e-947a-b134742e33d0" path="/var/lib/kubelet/pods/5bea82b3-f00f-450e-947a-b134742e33d0/volumes" Nov 02 07:45:36.678850 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:36.678811 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:36.824754 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:36.824723 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:37.644665 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:37.644583 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:38.435253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.435212 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:38.577769 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.577728 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt"] Nov 02 07:45:38.578388 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578372 5899 cpu_manager.go:414] "containerMap: removing container" podUID="376db59c-582f-400a-a63c-45811479e258" containerName="cleanup" Nov 02 07:45:38.578447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578390 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376db59c-582f-400a-a63c-45811479e258" containerName="cleanup" Nov 02 07:45:38.578447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578408 5899 cpu_manager.go:414] "containerMap: removing container" podUID="376db59c-582f-400a-a63c-45811479e258" containerName="cleanup" Nov 02 07:45:38.578447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578414 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376db59c-582f-400a-a63c-45811479e258" containerName="cleanup" Nov 02 07:45:38.578447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578424 5899 cpu_manager.go:414] "containerMap: removing container" podUID="376db59c-582f-400a-a63c-45811479e258" containerName="cleanup" Nov 02 07:45:38.578447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578430 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376db59c-582f-400a-a63c-45811479e258" containerName="cleanup" Nov 02 07:45:38.578447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578438 5899 cpu_manager.go:414] "containerMap: removing container" podUID="5bea82b3-f00f-450e-947a-b134742e33d0" containerName="maas-api" Nov 02 07:45:38.578447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578443 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bea82b3-f00f-450e-947a-b134742e33d0" containerName="maas-api" Nov 02 07:45:38.578656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578459 5899 cpu_manager.go:414] "containerMap: removing container" podUID="85e73a0c-facf-4baa-a3b7-48db4170f68f" containerName="cleanup" Nov 02 07:45:38.578656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578464 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e73a0c-facf-4baa-a3b7-48db4170f68f" containerName="cleanup" Nov 02 07:45:38.578656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578478 5899 cpu_manager.go:414] "containerMap: removing container" podUID="85e73a0c-facf-4baa-a3b7-48db4170f68f" containerName="cleanup" Nov 02 07:45:38.578656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578483 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e73a0c-facf-4baa-a3b7-48db4170f68f" containerName="cleanup" Nov 02 07:45:38.578656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578491 5899 cpu_manager.go:414] "containerMap: removing container" podUID="85e73a0c-facf-4baa-a3b7-48db4170f68f" containerName="cleanup" Nov 02 07:45:38.578656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578495 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e73a0c-facf-4baa-a3b7-48db4170f68f" containerName="cleanup" Nov 02 07:45:38.578656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578601 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="376db59c-582f-400a-a63c-45811479e258" containerName="cleanup" Nov 02 07:45:38.578656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578611 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="376db59c-582f-400a-a63c-45811479e258" containerName="cleanup" Nov 02 07:45:38.578656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578617 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="376db59c-582f-400a-a63c-45811479e258" containerName="cleanup" Nov 02 07:45:38.578656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578624 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bea82b3-f00f-450e-947a-b134742e33d0" containerName="maas-api" Nov 02 07:45:38.578656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578632 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e73a0c-facf-4baa-a3b7-48db4170f68f" containerName="cleanup" Nov 02 07:45:38.578656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578639 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e73a0c-facf-4baa-a3b7-48db4170f68f" containerName="cleanup" Nov 02 07:45:38.578656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.578643 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e73a0c-facf-4baa-a3b7-48db4170f68f" containerName="cleanup" Nov 02 07:45:38.579766 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.579747 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:38.582359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.582343 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" Nov 02 07:45:38.602375 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.602345 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt"] Nov 02 07:45:38.610703 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.610657 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dc62c2\"" Nov 02 07:45:38.610861 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.610702 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dc62c2-dockercfg-5rqxz\"" Nov 02 07:45:38.631941 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.631908 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk6qh\" (UniqueName: \"kubernetes.io/projected/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-kube-api-access-sk6qh\") pod \"payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt\" (UID: \"b9c2b99e-5043-4611-ba0a-aa67c5b85e74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" Nov 02 07:45:38.632240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.631995 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt\" (UID: \"b9c2b99e-5043-4611-ba0a-aa67c5b85e74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" Nov 02 07:45:38.661133 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.660411 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65"] Nov 02 07:45:38.666614 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.666569 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" Nov 02 07:45:38.705382 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.704228 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65"] Nov 02 07:45:38.737207 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.736304 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk6qh\" (UniqueName: \"kubernetes.io/projected/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-kube-api-access-sk6qh\") pod \"payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt\" (UID: \"b9c2b99e-5043-4611-ba0a-aa67c5b85e74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" Nov 02 07:45:38.737207 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.736395 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79ndn\" (UniqueName: \"kubernetes.io/projected/e9ea184a-196b-4077-83fb-53c52fa8024f-kube-api-access-79ndn\") pod \"payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65\" (UID: \"e9ea184a-196b-4077-83fb-53c52fa8024f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" Nov 02 07:45:38.737207 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.736473 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt\" (UID: \"b9c2b99e-5043-4611-ba0a-aa67c5b85e74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" Nov 02 07:45:38.737207 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.736508 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ea184a-196b-4077-83fb-53c52fa8024f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65\" (UID: \"e9ea184a-196b-4077-83fb-53c52fa8024f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" Nov 02 07:45:38.740605 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.738083 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt\" (UID: \"b9c2b99e-5043-4611-ba0a-aa67c5b85e74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" Nov 02 07:45:38.761274 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.761243 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk6qh\" (UniqueName: \"kubernetes.io/projected/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-kube-api-access-sk6qh\") pod \"payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt\" (UID: \"b9c2b99e-5043-4611-ba0a-aa67c5b85e74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" Nov 02 07:45:38.838373 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.838327 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79ndn\" (UniqueName: \"kubernetes.io/projected/e9ea184a-196b-4077-83fb-53c52fa8024f-kube-api-access-79ndn\") pod \"payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65\" (UID: \"e9ea184a-196b-4077-83fb-53c52fa8024f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" Nov 02 07:45:38.838579 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.838412 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ea184a-196b-4077-83fb-53c52fa8024f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65\" (UID: \"e9ea184a-196b-4077-83fb-53c52fa8024f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" Nov 02 07:45:38.839390 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.839360 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ea184a-196b-4077-83fb-53c52fa8024f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65\" (UID: \"e9ea184a-196b-4077-83fb-53c52fa8024f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" Nov 02 07:45:38.862861 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.862828 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79ndn\" (UniqueName: \"kubernetes.io/projected/e9ea184a-196b-4077-83fb-53c52fa8024f-kube-api-access-79ndn\") pod \"payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65\" (UID: \"e9ea184a-196b-4077-83fb-53c52fa8024f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" Nov 02 07:45:38.896476 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:38.896438 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" Nov 02 07:45:39.004101 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:39.004039 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" Nov 02 07:45:39.210712 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:39.210639 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt"] Nov 02 07:45:39.211375 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:45:39.211344 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c2b99e_5043_4611_ba0a_aa67c5b85e74.slice/crio-f4bc7a5d876d20990bb0760127f54dd524f4650cb23d62fa5b68e492793d006d WatchSource:0}: Error finding container f4bc7a5d876d20990bb0760127f54dd524f4650cb23d62fa5b68e492793d006d: Status 404 returned error can't find the container with id f4bc7a5d876d20990bb0760127f54dd524f4650cb23d62fa5b68e492793d006d Nov 02 07:45:39.321137 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:39.321108 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65"] Nov 02 07:45:39.322432 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:45:39.322398 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ea184a_196b_4077_83fb_53c52fa8024f.slice/crio-8157756d3bddd61187b9518eef2a723f5c7b43cdf6bcbbaa375bf89ef864c901 WatchSource:0}: Error finding container 8157756d3bddd61187b9518eef2a723f5c7b43cdf6bcbbaa375bf89ef864c901: Status 404 returned error can't find the container with id 8157756d3bddd61187b9518eef2a723f5c7b43cdf6bcbbaa375bf89ef864c901 Nov 02 07:45:39.754295 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:39.754259 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" event={"ID":"e9ea184a-196b-4077-83fb-53c52fa8024f","Type":"ContainerStarted","Data":"67d08e301aba46191059c182789f83b30673aacdc6c55abfd189e03d3918b0fa"} Nov 02 07:45:39.754805 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:39.754318 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" Nov 02 07:45:39.754805 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:39.754337 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" event={"ID":"e9ea184a-196b-4077-83fb-53c52fa8024f","Type":"ContainerStarted","Data":"8157756d3bddd61187b9518eef2a723f5c7b43cdf6bcbbaa375bf89ef864c901"} Nov 02 07:45:39.755984 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:39.755959 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" event={"ID":"b9c2b99e-5043-4611-ba0a-aa67c5b85e74","Type":"ContainerStarted","Data":"7d4361705ed6fcba8a7a392ea3b5f71c1f1cf9b97c5789bddaf99266322ecde1"} Nov 02 07:45:39.756126 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:39.756010 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" Nov 02 07:45:39.756126 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:39.756036 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" event={"ID":"b9c2b99e-5043-4611-ba0a-aa67c5b85e74","Type":"ContainerStarted","Data":"f4bc7a5d876d20990bb0760127f54dd524f4650cb23d62fa5b68e492793d006d"} Nov 02 07:45:39.835422 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:39.835363 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" podStartSLOduration=1.835348794 podStartE2EDuration="1.835348794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:39.831640388 +0000 UTC m=+1240.129921213" watchObservedRunningTime="2026-11-02 07:45:39.835348794 +0000 UTC m=+1240.133629810" Nov 02 07:45:39.877963 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:39.877911 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" podStartSLOduration=1.877897039 podStartE2EDuration="1.877897039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:39.873295327 +0000 UTC m=+1240.171576153" watchObservedRunningTime="2026-11-02 07:45:39.877897039 +0000 UTC m=+1240.176177863" Nov 02 07:45:40.391271 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:40.391238 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:42.695983 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:42.695944 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:43.141179 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:45:43.141134 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:45:43.534912 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:43.534879 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:43.682077 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:43.682040 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:44.320427 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:44.320383 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:44.634821 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:44.634739 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:44.776189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:44.776128 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:45.213315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:45.213277 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:45.360680 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:45.360625 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:46.600777 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:46.600747 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:46.970746 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:46.970648 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:48.428404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:48.428355 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:50.324522 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:50.324482 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:50.765413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:50.765382 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" Nov 02 07:45:50.765582 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:50.765459 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" Nov 02 07:45:52.529040 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.529007 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg"] Nov 02 07:45:52.533328 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.533306 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.547594 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.547571 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-54fd3c-serving-cert\"" Nov 02 07:45:52.563385 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.563337 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg"] Nov 02 07:45:52.635370 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.635332 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2d7w\" (UniqueName: \"kubernetes.io/projected/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-kube-api-access-n2d7w\") pod \"maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.635539 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.635391 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-openshift-service-ca\") pod \"maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.635539 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.635506 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-metrics-certs\") pod \"maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.635609 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.635543 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-maas-api-tls\") pod \"maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.643302 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.643271 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:52.736918 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.736877 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2d7w\" (UniqueName: \"kubernetes.io/projected/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-kube-api-access-n2d7w\") pod \"maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.737112 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.736949 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-openshift-service-ca\") pod \"maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.737112 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.737037 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-metrics-certs\") pod \"maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.737112 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.737094 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-maas-api-tls\") pod \"maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.737716 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.737692 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-openshift-service-ca\") pod \"maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.739717 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.739690 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-metrics-certs\") pod \"maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.739717 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.739708 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-maas-api-tls\") pod \"maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.762928 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.762902 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2d7w\" (UniqueName: \"kubernetes.io/projected/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-kube-api-access-n2d7w\") pod \"maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.782815 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.782740 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:52.847596 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.847559 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:52.929857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:52.929824 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:53.162354 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:53.162325 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg"] Nov 02 07:45:53.163156 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:45:53.163129 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a6b79c_ecb0_43cd_b8b8_00e4922f01c2.slice/crio-1d15558202e08940d796afdfb80c0fb9ee1ee225130a077ac13e4bc39e36f2ad WatchSource:0}: Error finding container 1d15558202e08940d796afdfb80c0fb9ee1ee225130a077ac13e4bc39e36f2ad: Status 404 returned error can't find the container with id 1d15558202e08940d796afdfb80c0fb9ee1ee225130a077ac13e4bc39e36f2ad Nov 02 07:45:53.866062 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:53.865918 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" event={"ID":"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2","Type":"ContainerStarted","Data":"6f5e5e48f4864e691e5c3919a45143193aac32ee3f9246a496b92f4475b4f11e"} Nov 02 07:45:53.866062 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:53.865991 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" event={"ID":"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2","Type":"ContainerStarted","Data":"1d15558202e08940d796afdfb80c0fb9ee1ee225130a077ac13e4bc39e36f2ad"} Nov 02 07:45:53.867030 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:53.866988 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:45:53.912337 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:53.912258 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" podStartSLOduration=1.488514795 podStartE2EDuration="1.912237992s" totalImagesPullingTime="423.723197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:53.908919035 +0000 UTC m=+1254.207199861" watchObservedRunningTime="2026-11-02 07:45:53.912237992 +0000 UTC m=+1254.210518818" Nov 02 07:45:54.009314 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.009269 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:54.726059 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.726022 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx"] Nov 02 07:45:54.730498 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.730472 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:54.744760 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.744737 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b2cc1f-serving-cert\"" Nov 02 07:45:54.760317 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.760293 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx"] Nov 02 07:45:54.781196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.781127 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-metrics-certs\") pod \"maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:54.781369 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.781300 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:54.781369 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.781359 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a271e4-894f-44af-9786-ca603e3895e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:54.781572 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.781553 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv22d\" (UniqueName: \"kubernetes.io/projected/42a271e4-894f-44af-9786-ca603e3895e9-kube-api-access-zv22d\") pod \"maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:54.874088 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.874044 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:54.882767 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.882729 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv22d\" (UniqueName: \"kubernetes.io/projected/42a271e4-894f-44af-9786-ca603e3895e9-kube-api-access-zv22d\") pod \"maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:54.882920 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.882779 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-metrics-certs\") pod \"maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:54.882920 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.882892 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:54.883024 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.882946 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a271e4-894f-44af-9786-ca603e3895e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:54.883180 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:45:54.883143 5899 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-serving-cert: secret "maas-api-e2e-ipp-b-b2cc1f-serving-cert" not found Nov 02 07:45:54.883276 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:45:54.883259 5899 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-maas-api-tls podName:42a271e4-894f-44af-9786-ca603e3895e9 nodeName:}" failed. No retries permitted until 2026-11-02 07:45:55.383238955 +0000 UTC m=+1255.681519761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-maas-api-tls") pod "maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" (UID: "42a271e4-894f-44af-9786-ca603e3895e9") : secret "maas-api-e2e-ipp-b-b2cc1f-serving-cert" not found Nov 02 07:45:54.883653 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.883635 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a271e4-894f-44af-9786-ca603e3895e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:54.885370 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.885347 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-metrics-certs\") pod \"maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:54.907020 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:54.906990 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv22d\" (UniqueName: \"kubernetes.io/projected/42a271e4-894f-44af-9786-ca603e3895e9-kube-api-access-zv22d\") pod \"maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:55.013437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:55.013401 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:55.289374 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:55.289288 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:55.390037 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:55.390006 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:55.392521 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:55.392497 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:55.645564 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:55.645470 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:56.017221 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:56.017186 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx"] Nov 02 07:45:56.039253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:56.039131 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:56.323979 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:56.323865 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:56.889751 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:56.889714 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:45:56.902570 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:56.902525 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" event={"ID":"42a271e4-894f-44af-9786-ca603e3895e9","Type":"ContainerStarted","Data":"e0dcee5ac3fe6cca85493bb07d281eddba50a3345964975af466a8667f4fa8bf"} Nov 02 07:45:56.902786 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:56.902593 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" event={"ID":"42a271e4-894f-44af-9786-ca603e3895e9","Type":"ContainerStarted","Data":"e92b2322b7f35efac778e278f93f9e18f590b390eb44da2399327f013ea62f7e"} Nov 02 07:45:56.902875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:56.902822 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:45:56.945771 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:56.945708 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" podStartSLOduration=2.641935878 podStartE2EDuration="2.945689733s" totalImagesPullingTime="303.753855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:56.944005778 +0000 UTC m=+1257.242286603" watchObservedRunningTime="2026-11-02 07:45:56.945689733 +0000 UTC m=+1257.243970559" Nov 02 07:45:58.678795 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:45:58.678759 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:00.525987 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:00.525951 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:00.885072 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:00.884994 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:46:02.806751 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:02.806711 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6"] Nov 02 07:46:02.812399 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:02.812374 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:02.840989 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:02.840957 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-dc62c2-serving-cert\"" Nov 02 07:46:02.850062 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:02.850028 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6"] Nov 02 07:46:02.910500 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:02.910460 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:02.910686 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:02.910598 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-metrics-certs\") pod \"maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:02.910750 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:02.910716 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cf5af6-6278-476a-b0ec-d3c91e1693ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:02.910825 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:02.910811 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf8hm\" (UniqueName: \"kubernetes.io/projected/81cf5af6-6278-476a-b0ec-d3c91e1693ee-kube-api-access-gf8hm\") pod \"maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:02.914991 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:02.914965 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:46:02.930485 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:02.930458 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:03.012191 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.012135 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cf5af6-6278-476a-b0ec-d3c91e1693ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:03.012389 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.012264 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf8hm\" (UniqueName: \"kubernetes.io/projected/81cf5af6-6278-476a-b0ec-d3c91e1693ee-kube-api-access-gf8hm\") pod \"maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:03.012501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.012475 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:03.012660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.012555 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-metrics-certs\") pod \"maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:03.012927 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.012904 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cf5af6-6278-476a-b0ec-d3c91e1693ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:03.015832 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.015803 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-metrics-certs\") pod \"maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:03.015938 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.015894 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:03.038495 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.038461 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf8hm\" (UniqueName: \"kubernetes.io/projected/81cf5af6-6278-476a-b0ec-d3c91e1693ee-kube-api-access-gf8hm\") pod \"maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:03.064589 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.064511 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:03.126197 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.126154 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:03.200419 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.200379 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:03.440693 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.440669 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6"] Nov 02 07:46:03.441692 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:46:03.441667 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cf5af6_6278_476a_b0ec_d3c91e1693ee.slice/crio-b1105904de0a4084962d354572c4f94f752da6f4107b4198aa6bbe3da7a5acec WatchSource:0}: Error finding container b1105904de0a4084962d354572c4f94f752da6f4107b4198aa6bbe3da7a5acec: Status 404 returned error can't find the container with id b1105904de0a4084962d354572c4f94f752da6f4107b4198aa6bbe3da7a5acec Nov 02 07:46:03.948779 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.948738 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" event={"ID":"81cf5af6-6278-476a-b0ec-d3c91e1693ee","Type":"ContainerStarted","Data":"65858e5762dc655308f8caadd28f996e45bc40006dab909f336a123e2fa0dcff"} Nov 02 07:46:03.949278 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.948807 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" event={"ID":"81cf5af6-6278-476a-b0ec-d3c91e1693ee","Type":"ContainerStarted","Data":"b1105904de0a4084962d354572c4f94f752da6f4107b4198aa6bbe3da7a5acec"} Nov 02 07:46:03.949278 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.948842 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:03.993265 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:03.993207 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" podStartSLOduration=1.637948416 podStartE2EDuration="1.993193936s" totalImagesPullingTime="355.24552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:46:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:46:03.989964137 +0000 UTC m=+1264.288244963" watchObservedRunningTime="2026-11-02 07:46:03.993193936 +0000 UTC m=+1264.291474755" Nov 02 07:46:04.896787 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:04.896757 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:06.447996 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:06.447953 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:06.646021 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:06.645981 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:06.791560 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:06.791532 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:07.342308 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:07.342263 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:09.063601 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:09.063559 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:09.206296 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:09.206262 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:09.693422 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:09.693390 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:09.923657 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:09.923623 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:09.961057 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:09.960955 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:46:10.306922 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:10.306891 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:11.247073 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:11.247035 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:11.820025 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:11.819968 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:11.964837 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:11.964800 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:13.142180 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:46:13.142130 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:46:13.657944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:13.657909 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:15.375143 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:15.375111 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:15.729021 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:15.728976 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:17.519309 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:17.519273 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:17.663368 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:17.663337 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:19.990334 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:19.990301 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:21.853665 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:21.853616 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:22.111010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:22.110917 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:23.799476 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:23.799436 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:24.188200 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:24.188100 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:24.409084 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:24.409050 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:25.769346 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:25.769314 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:26.089905 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:26.089829 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:27.884766 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:27.884731 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:29.768645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:29.768613 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:29.901531 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:29.901473 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:30.040921 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:30.040840 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:30.181577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:30.181546 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:31.854263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:31.854216 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:32.004309 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:32.004269 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:32.596360 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:32.596319 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:32.739686 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:32.739648 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:34.526125 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:34.526095 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:35.400679 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:35.400649 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78"] Nov 02 07:46:35.400902 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:35.400877 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" podUID="34975296-bf4a-4c2c-a4b2-0719760298da" containerName="istio-proxy" containerID="cri-o://1ba82a12e60340b97a9811df94c2f6deb3f689052457c2f7af3c729033197312" gracePeriod=30 Nov 02 07:46:36.766550 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:36.766517 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:37.676403 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:37.676295 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65"] Nov 02 07:46:37.706290 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:37.706256 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt"] Nov 02 07:46:37.879314 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:37.879272 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:38.033392 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:38.033356 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:38.436597 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:38.436494 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:39.325381 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:39.325349 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:39.475849 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:39.475787 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:40.672519 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.672453 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:46:40.820014 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.819973 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34975296-bf4a-4c2c-a4b2-0719760298da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34975296-bf4a-4c2c-a4b2-0719760298da-istio-podinfo\") pod \"34975296-bf4a-4c2c-a4b2-0719760298da\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " Nov 02 07:46:40.820246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.820043 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-credential-socket\") pod \"34975296-bf4a-4c2c-a4b2-0719760298da\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " Nov 02 07:46:40.820246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.820085 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-certs\") pod \"34975296-bf4a-4c2c-a4b2-0719760298da\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " Nov 02 07:46:40.820387 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.820244 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34975296-bf4a-4c2c-a4b2-0719760298da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34975296-bf4a-4c2c-a4b2-0719760298da-istiod-ca-cert\") pod \"34975296-bf4a-4c2c-a4b2-0719760298da\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " Nov 02 07:46:40.820387 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.820365 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-credential-socket" pod "34975296-bf4a-4c2c-a4b2-0719760298da" (UID: "34975296-bf4a-4c2c-a4b2-0719760298da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:46:40.820507 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.820480 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-certs" pod "34975296-bf4a-4c2c-a4b2-0719760298da" (UID: "34975296-bf4a-4c2c-a4b2-0719760298da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:46:40.820581 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.820489 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-kube-api-access-v9mtl\" (UniqueName: \"kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-kube-api-access-v9mtl\") pod \"34975296-bf4a-4c2c-a4b2-0719760298da\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " Nov 02 07:46:40.820581 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.820566 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-data\") pod \"34975296-bf4a-4c2c-a4b2-0719760298da\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " Nov 02 07:46:40.820667 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.820606 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-envoy\") pod \"34975296-bf4a-4c2c-a4b2-0719760298da\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " Nov 02 07:46:40.820667 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.820635 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-istio-token\" (UniqueName: \"kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-istio-token\") pod \"34975296-bf4a-4c2c-a4b2-0719760298da\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " Nov 02 07:46:40.820667 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.820633 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34975296-bf4a-4c2c-a4b2-0719760298da-istiod-ca-cert" pod "34975296-bf4a-4c2c-a4b2-0719760298da" (UID: "34975296-bf4a-4c2c-a4b2-0719760298da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:46:40.820827 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.820681 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-socket\") pod \"34975296-bf4a-4c2c-a4b2-0719760298da\" (UID: \"34975296-bf4a-4c2c-a4b2-0719760298da\") " Nov 02 07:46:40.820885 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.820849 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-data" pod "34975296-bf4a-4c2c-a4b2-0719760298da" (UID: "34975296-bf4a-4c2c-a4b2-0719760298da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:46:40.821071 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.821039 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-socket" pod "34975296-bf4a-4c2c-a4b2-0719760298da" (UID: "34975296-bf4a-4c2c-a4b2-0719760298da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:46:40.821351 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.821334 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:46:40.821445 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.821354 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:46:40.821445 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.821368 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34975296-bf4a-4c2c-a4b2-0719760298da-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:46:40.821445 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.821382 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:46:40.821445 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.821393 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:46:40.822552 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.822527 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-kube-api-access-v9mtl" pod "34975296-bf4a-4c2c-a4b2-0719760298da" (UID: "34975296-bf4a-4c2c-a4b2-0719760298da"). InnerVolumeSpecName "kube-api-access-v9mtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:46:40.822637 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.822580 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34975296-bf4a-4c2c-a4b2-0719760298da-istio-podinfo" pod "34975296-bf4a-4c2c-a4b2-0719760298da" (UID: "34975296-bf4a-4c2c-a4b2-0719760298da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:46:40.823042 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.823024 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-envoy" pod "34975296-bf4a-4c2c-a4b2-0719760298da" (UID: "34975296-bf4a-4c2c-a4b2-0719760298da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:46:40.823100 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.823043 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-istio-token" pod "34975296-bf4a-4c2c-a4b2-0719760298da" (UID: "34975296-bf4a-4c2c-a4b2-0719760298da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:46:40.922895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.922806 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9mtl\" (UniqueName: \"kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-kube-api-access-v9mtl\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:46:40.922895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.922838 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34975296-bf4a-4c2c-a4b2-0719760298da-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:46:40.922895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.922848 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34975296-bf4a-4c2c-a4b2-0719760298da-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:46:40.922895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:40.922856 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34975296-bf4a-4c2c-a4b2-0719760298da-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:46:41.253482 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:41.253446 5899 generic.go:356] "Generic (PLEG): container finished" podID="34975296-bf4a-4c2c-a4b2-0719760298da" containerID="1ba82a12e60340b97a9811df94c2f6deb3f689052457c2f7af3c729033197312" exitCode=0 Nov 02 07:46:41.253655 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:41.253527 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" Nov 02 07:46:41.253655 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:41.253536 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" event={"ID":"34975296-bf4a-4c2c-a4b2-0719760298da","Type":"ContainerDied","Data":"1ba82a12e60340b97a9811df94c2f6deb3f689052457c2f7af3c729033197312"} Nov 02 07:46:41.253655 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:41.253627 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78" event={"ID":"34975296-bf4a-4c2c-a4b2-0719760298da","Type":"ContainerDied","Data":"9f97c484b84daa3eeacb8b99d68f011d267cd0a4b219f3a25ded9925224b68a9"} Nov 02 07:46:41.253655 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:41.253652 5899 scope.go:122] "RemoveContainer" containerID="1ba82a12e60340b97a9811df94c2f6deb3f689052457c2f7af3c729033197312" Nov 02 07:46:41.256439 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:41.256414 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:41.270145 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:41.269061 5899 scope.go:122] "RemoveContainer" containerID="1ba82a12e60340b97a9811df94c2f6deb3f689052457c2f7af3c729033197312" Nov 02 07:46:41.270145 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:46:41.269458 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba82a12e60340b97a9811df94c2f6deb3f689052457c2f7af3c729033197312\": container with ID starting with 1ba82a12e60340b97a9811df94c2f6deb3f689052457c2f7af3c729033197312 not found: ID does not exist" containerID="1ba82a12e60340b97a9811df94c2f6deb3f689052457c2f7af3c729033197312" Nov 02 07:46:41.270145 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:41.269492 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba82a12e60340b97a9811df94c2f6deb3f689052457c2f7af3c729033197312"} err="failed to get container status \"1ba82a12e60340b97a9811df94c2f6deb3f689052457c2f7af3c729033197312\": rpc error: code = NotFound desc = could not find container \"1ba82a12e60340b97a9811df94c2f6deb3f689052457c2f7af3c729033197312\": container with ID starting with 1ba82a12e60340b97a9811df94c2f6deb3f689052457c2f7af3c729033197312 not found: ID does not exist" Nov 02 07:46:41.315190 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:41.307207 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78"] Nov 02 07:46:41.333338 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:41.332738 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-txc78"] Nov 02 07:46:41.821349 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:41.821314 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34975296-bf4a-4c2c-a4b2-0719760298da" path="/var/lib/kubelet/pods/34975296-bf4a-4c2c-a4b2-0719760298da/volumes" Nov 02 07:46:42.677415 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:42.677372 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" podUID="e9ea184a-196b-4077-83fb-53c52fa8024f" containerName="payload-processing" containerID="cri-o://67d08e301aba46191059c182789f83b30673aacdc6c55abfd189e03d3918b0fa" gracePeriod=25 Nov 02 07:46:42.706634 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:42.706583 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" podUID="b9c2b99e-5043-4611-ba0a-aa67c5b85e74" containerName="payload-pre-processing" containerID="cri-o://7d4361705ed6fcba8a7a392ea3b5f71c1f1cf9b97c5789bddaf99266322ecde1" gracePeriod=25 Nov 02 07:46:42.803506 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:42.803472 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:42.998210 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:42.996692 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:43.071947 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.071921 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" Nov 02 07:46:43.075656 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.075632 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" Nov 02 07:46:43.141740 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.141690 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:43.150325 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:46:43.150286 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:46:43.162875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.162846 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ea184a-196b-4077-83fb-53c52fa8024f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ea184a-196b-4077-83fb-53c52fa8024f-plugins-config-volume\") pod \"e9ea184a-196b-4077-83fb-53c52fa8024f\" (UID: \"e9ea184a-196b-4077-83fb-53c52fa8024f\") " Nov 02 07:46:43.163053 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.162885 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-plugins-config-volume\") pod \"b9c2b99e-5043-4611-ba0a-aa67c5b85e74\" (UID: \"b9c2b99e-5043-4611-ba0a-aa67c5b85e74\") " Nov 02 07:46:43.163053 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.162941 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-kube-api-access-sk6qh\" (UniqueName: \"kubernetes.io/projected/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-kube-api-access-sk6qh\") pod \"b9c2b99e-5043-4611-ba0a-aa67c5b85e74\" (UID: \"b9c2b99e-5043-4611-ba0a-aa67c5b85e74\") " Nov 02 07:46:43.163195 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.163122 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ea184a-196b-4077-83fb-53c52fa8024f-kube-api-access-79ndn\" (UniqueName: \"kubernetes.io/projected/e9ea184a-196b-4077-83fb-53c52fa8024f-kube-api-access-79ndn\") pod \"e9ea184a-196b-4077-83fb-53c52fa8024f\" (UID: \"e9ea184a-196b-4077-83fb-53c52fa8024f\") " Nov 02 07:46:43.163411 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.163386 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-plugins-config-volume" pod "b9c2b99e-5043-4611-ba0a-aa67c5b85e74" (UID: "b9c2b99e-5043-4611-ba0a-aa67c5b85e74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:46:43.163819 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.163790 5899 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-plugins-config-volume\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:46:43.163934 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.163903 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ea184a-196b-4077-83fb-53c52fa8024f-plugins-config-volume" pod "e9ea184a-196b-4077-83fb-53c52fa8024f" (UID: "e9ea184a-196b-4077-83fb-53c52fa8024f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:46:43.165687 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.165651 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-kube-api-access-sk6qh" pod "b9c2b99e-5043-4611-ba0a-aa67c5b85e74" (UID: "b9c2b99e-5043-4611-ba0a-aa67c5b85e74"). InnerVolumeSpecName "kube-api-access-sk6qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:46:43.166198 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.166153 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ea184a-196b-4077-83fb-53c52fa8024f-kube-api-access-79ndn" pod "e9ea184a-196b-4077-83fb-53c52fa8024f" (UID: "e9ea184a-196b-4077-83fb-53c52fa8024f"). InnerVolumeSpecName "kube-api-access-79ndn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:46:43.265946 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.265914 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79ndn\" (UniqueName: \"kubernetes.io/projected/e9ea184a-196b-4077-83fb-53c52fa8024f-kube-api-access-79ndn\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:46:43.265946 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.265945 5899 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ea184a-196b-4077-83fb-53c52fa8024f-plugins-config-volume\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:46:43.265946 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.265961 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk6qh\" (UniqueName: \"kubernetes.io/projected/b9c2b99e-5043-4611-ba0a-aa67c5b85e74-kube-api-access-sk6qh\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:46:43.284292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.282562 5899 generic.go:356] "Generic (PLEG): container finished" podID="e9ea184a-196b-4077-83fb-53c52fa8024f" containerID="67d08e301aba46191059c182789f83b30673aacdc6c55abfd189e03d3918b0fa" exitCode=0 Nov 02 07:46:43.284292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.282719 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" event={"ID":"e9ea184a-196b-4077-83fb-53c52fa8024f","Type":"ContainerDied","Data":"67d08e301aba46191059c182789f83b30673aacdc6c55abfd189e03d3918b0fa"} Nov 02 07:46:43.284292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.282776 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" event={"ID":"e9ea184a-196b-4077-83fb-53c52fa8024f","Type":"ContainerDied","Data":"8157756d3bddd61187b9518eef2a723f5c7b43cdf6bcbbaa375bf89ef864c901"} Nov 02 07:46:43.284292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.282807 5899 scope.go:122] "RemoveContainer" containerID="67d08e301aba46191059c182789f83b30673aacdc6c55abfd189e03d3918b0fa" Nov 02 07:46:43.284292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.282913 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65" Nov 02 07:46:43.322289 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.322222 5899 generic.go:356] "Generic (PLEG): container finished" podID="b9c2b99e-5043-4611-ba0a-aa67c5b85e74" containerID="7d4361705ed6fcba8a7a392ea3b5f71c1f1cf9b97c5789bddaf99266322ecde1" exitCode=0 Nov 02 07:46:43.322424 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.322321 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" event={"ID":"b9c2b99e-5043-4611-ba0a-aa67c5b85e74","Type":"ContainerDied","Data":"7d4361705ed6fcba8a7a392ea3b5f71c1f1cf9b97c5789bddaf99266322ecde1"} Nov 02 07:46:43.322424 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.322372 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" event={"ID":"b9c2b99e-5043-4611-ba0a-aa67c5b85e74","Type":"ContainerDied","Data":"f4bc7a5d876d20990bb0760127f54dd524f4650cb23d62fa5b68e492793d006d"} Nov 02 07:46:43.322578 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.322452 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt" Nov 02 07:46:43.375905 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.375870 5899 scope.go:122] "RemoveContainer" containerID="67d08e301aba46191059c182789f83b30673aacdc6c55abfd189e03d3918b0fa" Nov 02 07:46:43.376428 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:46:43.376399 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d08e301aba46191059c182789f83b30673aacdc6c55abfd189e03d3918b0fa\": container with ID starting with 67d08e301aba46191059c182789f83b30673aacdc6c55abfd189e03d3918b0fa not found: ID does not exist" containerID="67d08e301aba46191059c182789f83b30673aacdc6c55abfd189e03d3918b0fa" Nov 02 07:46:43.376531 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.376443 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d08e301aba46191059c182789f83b30673aacdc6c55abfd189e03d3918b0fa"} err="failed to get container status \"67d08e301aba46191059c182789f83b30673aacdc6c55abfd189e03d3918b0fa\": rpc error: code = NotFound desc = could not find container \"67d08e301aba46191059c182789f83b30673aacdc6c55abfd189e03d3918b0fa\": container with ID starting with 67d08e301aba46191059c182789f83b30673aacdc6c55abfd189e03d3918b0fa not found: ID does not exist" Nov 02 07:46:43.376531 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.376470 5899 scope.go:122] "RemoveContainer" containerID="7d4361705ed6fcba8a7a392ea3b5f71c1f1cf9b97c5789bddaf99266322ecde1" Nov 02 07:46:43.387917 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.387892 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt"] Nov 02 07:46:43.391634 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.391607 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc62c2-56869f88f5-jmjlt"] Nov 02 07:46:43.397145 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.397116 5899 scope.go:122] "RemoveContainer" containerID="7d4361705ed6fcba8a7a392ea3b5f71c1f1cf9b97c5789bddaf99266322ecde1" Nov 02 07:46:43.397551 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:46:43.397530 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4361705ed6fcba8a7a392ea3b5f71c1f1cf9b97c5789bddaf99266322ecde1\": container with ID starting with 7d4361705ed6fcba8a7a392ea3b5f71c1f1cf9b97c5789bddaf99266322ecde1 not found: ID does not exist" containerID="7d4361705ed6fcba8a7a392ea3b5f71c1f1cf9b97c5789bddaf99266322ecde1" Nov 02 07:46:43.397629 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.397558 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4361705ed6fcba8a7a392ea3b5f71c1f1cf9b97c5789bddaf99266322ecde1"} err="failed to get container status \"7d4361705ed6fcba8a7a392ea3b5f71c1f1cf9b97c5789bddaf99266322ecde1\": rpc error: code = NotFound desc = could not find container \"7d4361705ed6fcba8a7a392ea3b5f71c1f1cf9b97c5789bddaf99266322ecde1\": container with ID starting with 7d4361705ed6fcba8a7a392ea3b5f71c1f1cf9b97c5789bddaf99266322ecde1 not found: ID does not exist" Nov 02 07:46:43.424412 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.424375 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65"] Nov 02 07:46:43.428862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.428827 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc62c2-8fdc4fdff-cjb65"] Nov 02 07:46:43.766445 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.766397 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:43.823947 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.823913 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c2b99e-5043-4611-ba0a-aa67c5b85e74" path="/var/lib/kubelet/pods/b9c2b99e-5043-4611-ba0a-aa67c5b85e74/volumes" Nov 02 07:46:43.824433 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:43.824417 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ea184a-196b-4077-83fb-53c52fa8024f" path="/var/lib/kubelet/pods/e9ea184a-196b-4077-83fb-53c52fa8024f/volumes" Nov 02 07:46:45.680349 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:45.680313 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:45.823133 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:45.823091 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:46.597985 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:46.597950 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:46.810410 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:46.810365 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:46.962457 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:46.962370 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:47.096607 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:47.096570 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:47.283375 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:47.283341 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:47.503079 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:47.503043 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:47.640790 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:47.640706 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:47.784732 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:47.784698 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:48.039234 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:48.039186 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:48.793953 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:48.793918 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:49.462266 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:49.462230 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:49.931937 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:49.931900 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:50.086182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:50.083018 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:50.798793 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:50.798758 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:52.766192 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:52.766106 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:53.745363 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:53.745332 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:53.890857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:53.890827 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:54.044854 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:54.044741 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:54.892242 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:54.892211 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:55.550258 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:55.550227 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:55.710349 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:55.710317 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:56.953611 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:56.953579 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:57.119409 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:57.119375 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:46:58.222400 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:46:58.222361 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:00.147922 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:00.147884 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:00.624001 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:00.623945 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh"] Nov 02 07:47:00.624301 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:00.624198 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" podUID="1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" containerName="istio-proxy" containerID="cri-o://7a2592e89229dd181f51f0d64b85b9bb12283be0db710adab7ab48057decad08" gracePeriod=30 Nov 02 07:47:01.244947 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:01.244899 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:02.387113 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:02.387083 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:02.652587 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:02.652512 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:03.042471 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:03.042441 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:03.190101 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:03.190041 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:04.111304 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:04.111272 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:05.816198 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:05.815069 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:05.931199 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:05.931172 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:47:06.077617 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.077580 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-podinfo\") pod \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " Nov 02 07:47:06.077784 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.077665 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-credential-socket\") pod \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " Nov 02 07:47:06.077784 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.077737 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-kube-api-access-6cfj6\" (UniqueName: \"kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-kube-api-access-6cfj6\") pod \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " Nov 02 07:47:06.077784 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.077769 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istiod-ca-cert\") pod \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " Nov 02 07:47:06.077911 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.077886 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-socket\") pod \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " Nov 02 07:47:06.077960 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.077938 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-data\") pod \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " Nov 02 07:47:06.078010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.077983 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-certs\") pod \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " Nov 02 07:47:06.078070 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.078006 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-credential-socket" pod "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" (UID: "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:47:06.078183 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.078149 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-envoy\") pod \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " Nov 02 07:47:06.078253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.078212 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-socket" pod "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" (UID: "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:47:06.078253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.078224 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-token\" (UniqueName: \"kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-token\") pod \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\" (UID: \"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8\") " Nov 02 07:47:06.078253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.078229 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-data" pod "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" (UID: "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:47:06.078397 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.078247 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-certs" pod "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" (UID: "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:47:06.078397 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.078278 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istiod-ca-cert" pod "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" (UID: "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:47:06.078855 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.078835 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:06.078949 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.078857 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:06.078949 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.078869 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:06.078949 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.078880 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:06.078949 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.078891 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:06.079910 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.079892 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-podinfo" pod "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" (UID: "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:47:06.080150 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.080129 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-token" pod "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" (UID: "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:47:06.080554 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.080533 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-envoy" pod "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" (UID: "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:47:06.080614 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.080553 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-kube-api-access-6cfj6" pod "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" (UID: "1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8"). InnerVolumeSpecName "kube-api-access-6cfj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:47:06.180101 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.180022 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:06.180101 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.180052 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cfj6\" (UniqueName: \"kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-kube-api-access-6cfj6\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:06.180101 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.180062 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:06.180101 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.180071 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:06.483353 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.483316 5899 generic.go:356] "Generic (PLEG): container finished" podID="1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" containerID="7a2592e89229dd181f51f0d64b85b9bb12283be0db710adab7ab48057decad08" exitCode=0 Nov 02 07:47:06.483524 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.483373 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" Nov 02 07:47:06.483524 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.483378 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" event={"ID":"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8","Type":"ContainerDied","Data":"7a2592e89229dd181f51f0d64b85b9bb12283be0db710adab7ab48057decad08"} Nov 02 07:47:06.483524 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.483430 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh" event={"ID":"1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8","Type":"ContainerDied","Data":"851121003da019a99fddfc3c5beabdf62fcd2e3b3333e1cf0029ac9b1c52c4e2"} Nov 02 07:47:06.483524 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.483462 5899 scope.go:122] "RemoveContainer" containerID="7a2592e89229dd181f51f0d64b85b9bb12283be0db710adab7ab48057decad08" Nov 02 07:47:06.496138 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.496113 5899 scope.go:122] "RemoveContainer" containerID="7a2592e89229dd181f51f0d64b85b9bb12283be0db710adab7ab48057decad08" Nov 02 07:47:06.497302 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:47:06.497276 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2592e89229dd181f51f0d64b85b9bb12283be0db710adab7ab48057decad08\": container with ID starting with 7a2592e89229dd181f51f0d64b85b9bb12283be0db710adab7ab48057decad08 not found: ID does not exist" containerID="7a2592e89229dd181f51f0d64b85b9bb12283be0db710adab7ab48057decad08" Nov 02 07:47:06.497387 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.497309 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2592e89229dd181f51f0d64b85b9bb12283be0db710adab7ab48057decad08"} err="failed to get container status \"7a2592e89229dd181f51f0d64b85b9bb12283be0db710adab7ab48057decad08\": rpc error: code = NotFound desc = could not find container \"7a2592e89229dd181f51f0d64b85b9bb12283be0db710adab7ab48057decad08\": container with ID starting with 7a2592e89229dd181f51f0d64b85b9bb12283be0db710adab7ab48057decad08 not found: ID does not exist" Nov 02 07:47:06.547031 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.546991 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh"] Nov 02 07:47:06.559859 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:06.559821 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-d74vh"] Nov 02 07:47:07.180700 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:07.180664 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg"] Nov 02 07:47:07.277866 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:07.277833 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:07.510942 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:07.510878 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" podUID="64a6b79c-ecb0-43cd-b8b8-00e4922f01c2" containerName="maas-api" containerID="cri-o://6f5e5e48f4864e691e5c3919a45143193aac32ee3f9246a496b92f4475b4f11e" gracePeriod=30 Nov 02 07:47:07.820711 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:07.820673 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" path="/var/lib/kubelet/pods/1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8/volumes" Nov 02 07:47:07.840863 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:07.840838 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:47:08.010988 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.010953 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-kube-api-access-n2d7w\" (UniqueName: \"kubernetes.io/projected/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-kube-api-access-n2d7w\") pod \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " Nov 02 07:47:08.011170 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.011082 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-maas-api-tls\") pod \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " Nov 02 07:47:08.011309 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.011294 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-openshift-service-ca\") pod \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " Nov 02 07:47:08.011343 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.011319 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-metrics-certs\") pod \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\" (UID: \"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2\") " Nov 02 07:47:08.011707 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.011678 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-openshift-service-ca" pod "64a6b79c-ecb0-43cd-b8b8-00e4922f01c2" (UID: "64a6b79c-ecb0-43cd-b8b8-00e4922f01c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:47:08.011884 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.011868 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:08.013214 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.013153 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-maas-api-tls" pod "64a6b79c-ecb0-43cd-b8b8-00e4922f01c2" (UID: "64a6b79c-ecb0-43cd-b8b8-00e4922f01c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:47:08.013314 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.013208 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-kube-api-access-n2d7w" pod "64a6b79c-ecb0-43cd-b8b8-00e4922f01c2" (UID: "64a6b79c-ecb0-43cd-b8b8-00e4922f01c2"). InnerVolumeSpecName "kube-api-access-n2d7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:47:08.013314 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.013247 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-metrics-certs" pod "64a6b79c-ecb0-43cd-b8b8-00e4922f01c2" (UID: "64a6b79c-ecb0-43cd-b8b8-00e4922f01c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:47:08.113867 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.113787 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:08.113867 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.113817 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:08.113867 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.113826 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2d7w\" (UniqueName: \"kubernetes.io/projected/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2-kube-api-access-n2d7w\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:08.135433 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.135398 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:08.518421 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.518387 5899 generic.go:356] "Generic (PLEG): container finished" podID="64a6b79c-ecb0-43cd-b8b8-00e4922f01c2" containerID="6f5e5e48f4864e691e5c3919a45143193aac32ee3f9246a496b92f4475b4f11e" exitCode=0 Nov 02 07:47:08.518809 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.518455 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" Nov 02 07:47:08.518809 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.518468 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" event={"ID":"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2","Type":"ContainerDied","Data":"6f5e5e48f4864e691e5c3919a45143193aac32ee3f9246a496b92f4475b4f11e"} Nov 02 07:47:08.518809 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.518553 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg" event={"ID":"64a6b79c-ecb0-43cd-b8b8-00e4922f01c2","Type":"ContainerDied","Data":"1d15558202e08940d796afdfb80c0fb9ee1ee225130a077ac13e4bc39e36f2ad"} Nov 02 07:47:08.518809 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.518582 5899 scope.go:122] "RemoveContainer" containerID="6f5e5e48f4864e691e5c3919a45143193aac32ee3f9246a496b92f4475b4f11e" Nov 02 07:47:08.529827 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.529802 5899 scope.go:122] "RemoveContainer" containerID="6f5e5e48f4864e691e5c3919a45143193aac32ee3f9246a496b92f4475b4f11e" Nov 02 07:47:08.530141 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:47:08.530110 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f5e5e48f4864e691e5c3919a45143193aac32ee3f9246a496b92f4475b4f11e\": container with ID starting with 6f5e5e48f4864e691e5c3919a45143193aac32ee3f9246a496b92f4475b4f11e not found: ID does not exist" containerID="6f5e5e48f4864e691e5c3919a45143193aac32ee3f9246a496b92f4475b4f11e" Nov 02 07:47:08.530233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.530149 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f5e5e48f4864e691e5c3919a45143193aac32ee3f9246a496b92f4475b4f11e"} err="failed to get container status \"6f5e5e48f4864e691e5c3919a45143193aac32ee3f9246a496b92f4475b4f11e\": rpc error: code = NotFound desc = could not find container \"6f5e5e48f4864e691e5c3919a45143193aac32ee3f9246a496b92f4475b4f11e\": container with ID starting with 6f5e5e48f4864e691e5c3919a45143193aac32ee3f9246a496b92f4475b4f11e not found: ID does not exist" Nov 02 07:47:08.570044 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.570011 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg"] Nov 02 07:47:08.581618 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:08.581587 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-54fd3c-79db47bfcb-9tdvg"] Nov 02 07:47:09.820677 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:09.820644 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a6b79c-ecb0-43cd-b8b8-00e4922f01c2" path="/var/lib/kubelet/pods/64a6b79c-ecb0-43cd-b8b8-00e4922f01c2/volumes" Nov 02 07:47:09.888688 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:09.888656 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:10.491131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:10.491094 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:12.203951 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:12.203907 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:13.142563 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:47:13.142526 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:47:13.950869 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:13.950833 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:14.113652 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:14.113615 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:14.248930 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:14.248896 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:15.451443 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.451408 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs"] Nov 02 07:47:15.451940 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.451667 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" podUID="3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d" containerName="maas-api" containerID="cri-o://4dae99236007216d9831e2798db3e8bad17c152826f1d1fc53673f4f42d663d7" gracePeriod=30 Nov 02 07:47:15.544603 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.544571 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:15.569139 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.569103 5899 generic.go:356] "Generic (PLEG): container finished" podID="3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d" containerID="4dae99236007216d9831e2798db3e8bad17c152826f1d1fc53673f4f42d663d7" exitCode=0 Nov 02 07:47:15.569277 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.569187 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" event={"ID":"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d","Type":"ContainerDied","Data":"4dae99236007216d9831e2798db3e8bad17c152826f1d1fc53673f4f42d663d7"} Nov 02 07:47:15.735564 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.735398 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:15.779112 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.779080 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:47:15.845210 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.845151 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-maas-api-tls\") pod \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " Nov 02 07:47:15.845416 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.845306 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-metrics-certs\") pod \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " Nov 02 07:47:15.845416 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.845357 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-kube-api-access-t5fl6\" (UniqueName: \"kubernetes.io/projected/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-kube-api-access-t5fl6\") pod \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " Nov 02 07:47:15.845559 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.845487 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-openshift-service-ca\") pod \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\" (UID: \"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d\") " Nov 02 07:47:15.846029 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.845993 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-openshift-service-ca" pod "3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d" (UID: "3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:47:15.846849 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.846825 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:15.848571 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.848542 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-metrics-certs" pod "3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d" (UID: "3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:47:15.849055 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.849017 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-kube-api-access-t5fl6" pod "3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d" (UID: "3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d"). InnerVolumeSpecName "kube-api-access-t5fl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:47:15.850108 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.850072 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-maas-api-tls" pod "3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d" (UID: "3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:47:15.868101 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.868074 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:15.948501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.948416 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:15.948501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.948447 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5fl6\" (UniqueName: \"kubernetes.io/projected/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-kube-api-access-t5fl6\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:15.948501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:15.948458 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:16.370522 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:16.370485 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:16.577623 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:16.577590 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" Nov 02 07:47:16.577623 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:16.577612 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs" event={"ID":"3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d","Type":"ContainerDied","Data":"fac7d6636d782e94d4f224aa8ff5c6667308fc6a5bfb03258b98d48b185eb893"} Nov 02 07:47:16.578147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:16.577707 5899 scope.go:122] "RemoveContainer" containerID="4dae99236007216d9831e2798db3e8bad17c152826f1d1fc53673f4f42d663d7" Nov 02 07:47:16.640397 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:16.640311 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs"] Nov 02 07:47:16.664798 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:16.664767 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-523de0-74ddf895c8-qh7fs"] Nov 02 07:47:16.673871 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:16.673832 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:16.837211 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:16.837178 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:17.359772 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:17.359742 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:17.733178 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:17.733139 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:17.860184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:17.857530 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d" path="/var/lib/kubelet/pods/3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d/volumes" Nov 02 07:47:18.010491 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:18.010407 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:18.369480 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:18.362518 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j"] Nov 02 07:47:18.369480 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:18.362770 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" podUID="09a70f3e-6112-4665-b57a-bbe22502fec3" containerName="istio-proxy" containerID="cri-o://9877ab6cd4ce17203c0669a68ac0d3eb9f69a383069aee4eb312030b9d724dbb" gracePeriod=30 Nov 02 07:47:18.420081 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:18.420031 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:18.675263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:18.675182 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:19.177227 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:19.177194 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:19.318645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:19.318615 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:19.471238 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:19.468753 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:21.054530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:21.054497 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:21.661121 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:21.661092 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:23.287190 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.287141 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:23.436658 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.436621 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:23.636828 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.636797 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:47:23.650211 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.650176 5899 generic.go:356] "Generic (PLEG): container finished" podID="09a70f3e-6112-4665-b57a-bbe22502fec3" containerID="9877ab6cd4ce17203c0669a68ac0d3eb9f69a383069aee4eb312030b9d724dbb" exitCode=0 Nov 02 07:47:23.650359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.650223 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" Nov 02 07:47:23.650359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.650213 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" event={"ID":"09a70f3e-6112-4665-b57a-bbe22502fec3","Type":"ContainerDied","Data":"9877ab6cd4ce17203c0669a68ac0d3eb9f69a383069aee4eb312030b9d724dbb"} Nov 02 07:47:23.650359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.650347 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j" event={"ID":"09a70f3e-6112-4665-b57a-bbe22502fec3","Type":"ContainerDied","Data":"7726f4a856691a1ea39b55cb1acd2f08f8e26e2ef8eadd0e2ae5dd6d0bd035e1"} Nov 02 07:47:23.650482 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.650370 5899 scope.go:122] "RemoveContainer" containerID="9877ab6cd4ce17203c0669a68ac0d3eb9f69a383069aee4eb312030b9d724dbb" Nov 02 07:47:23.659630 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.659613 5899 scope.go:122] "RemoveContainer" containerID="9877ab6cd4ce17203c0669a68ac0d3eb9f69a383069aee4eb312030b9d724dbb" Nov 02 07:47:23.659912 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:47:23.659894 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9877ab6cd4ce17203c0669a68ac0d3eb9f69a383069aee4eb312030b9d724dbb\": container with ID starting with 9877ab6cd4ce17203c0669a68ac0d3eb9f69a383069aee4eb312030b9d724dbb not found: ID does not exist" containerID="9877ab6cd4ce17203c0669a68ac0d3eb9f69a383069aee4eb312030b9d724dbb" Nov 02 07:47:23.659962 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.659921 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9877ab6cd4ce17203c0669a68ac0d3eb9f69a383069aee4eb312030b9d724dbb"} err="failed to get container status \"9877ab6cd4ce17203c0669a68ac0d3eb9f69a383069aee4eb312030b9d724dbb\": rpc error: code = NotFound desc = could not find container \"9877ab6cd4ce17203c0669a68ac0d3eb9f69a383069aee4eb312030b9d724dbb\": container with ID starting with 9877ab6cd4ce17203c0669a68ac0d3eb9f69a383069aee4eb312030b9d724dbb not found: ID does not exist" Nov 02 07:47:23.709781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.709722 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:23.764128 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.764088 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-envoy\") pod \"09a70f3e-6112-4665-b57a-bbe22502fec3\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " Nov 02 07:47:23.764128 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.764134 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-kube-api-access-pbgs9\" (UniqueName: \"kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-kube-api-access-pbgs9\") pod \"09a70f3e-6112-4665-b57a-bbe22502fec3\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " Nov 02 07:47:23.764438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.764202 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-data\") pod \"09a70f3e-6112-4665-b57a-bbe22502fec3\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " Nov 02 07:47:23.764438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.764254 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-socket\") pod \"09a70f3e-6112-4665-b57a-bbe22502fec3\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " Nov 02 07:47:23.764438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.764300 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-podinfo\") pod \"09a70f3e-6112-4665-b57a-bbe22502fec3\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " Nov 02 07:47:23.764438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.764348 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-certs\") pod \"09a70f3e-6112-4665-b57a-bbe22502fec3\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " Nov 02 07:47:23.764438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.764408 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-token\" (UniqueName: \"kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-token\") pod \"09a70f3e-6112-4665-b57a-bbe22502fec3\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " Nov 02 07:47:23.764816 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.764444 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a70f3e-6112-4665-b57a-bbe22502fec3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a70f3e-6112-4665-b57a-bbe22502fec3-istiod-ca-cert\") pod \"09a70f3e-6112-4665-b57a-bbe22502fec3\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " Nov 02 07:47:23.764816 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.764580 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-credential-socket\") pod \"09a70f3e-6112-4665-b57a-bbe22502fec3\" (UID: \"09a70f3e-6112-4665-b57a-bbe22502fec3\") " Nov 02 07:47:23.765265 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.765235 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-data" pod "09a70f3e-6112-4665-b57a-bbe22502fec3" (UID: "09a70f3e-6112-4665-b57a-bbe22502fec3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:47:23.766202 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.765708 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-socket" pod "09a70f3e-6112-4665-b57a-bbe22502fec3" (UID: "09a70f3e-6112-4665-b57a-bbe22502fec3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:47:23.766202 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.765841 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-certs" pod "09a70f3e-6112-4665-b57a-bbe22502fec3" (UID: "09a70f3e-6112-4665-b57a-bbe22502fec3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:47:23.766202 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.766152 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a70f3e-6112-4665-b57a-bbe22502fec3-istiod-ca-cert" pod "09a70f3e-6112-4665-b57a-bbe22502fec3" (UID: "09a70f3e-6112-4665-b57a-bbe22502fec3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:47:23.766741 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.766717 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-credential-socket" pod "09a70f3e-6112-4665-b57a-bbe22502fec3" (UID: "09a70f3e-6112-4665-b57a-bbe22502fec3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:47:23.767713 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.767684 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-podinfo" pod "09a70f3e-6112-4665-b57a-bbe22502fec3" (UID: "09a70f3e-6112-4665-b57a-bbe22502fec3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:47:23.768067 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.768043 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-token" pod "09a70f3e-6112-4665-b57a-bbe22502fec3" (UID: "09a70f3e-6112-4665-b57a-bbe22502fec3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:47:23.768413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.768391 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-kube-api-access-pbgs9" pod "09a70f3e-6112-4665-b57a-bbe22502fec3" (UID: "09a70f3e-6112-4665-b57a-bbe22502fec3"). InnerVolumeSpecName "kube-api-access-pbgs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:47:23.770736 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.770715 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-envoy" pod "09a70f3e-6112-4665-b57a-bbe22502fec3" (UID: "09a70f3e-6112-4665-b57a-bbe22502fec3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:47:23.867007 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.866934 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:23.867007 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.866960 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:23.867007 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.866970 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbgs9\" (UniqueName: \"kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-kube-api-access-pbgs9\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:23.867007 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.866981 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:23.867007 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.866991 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:23.867007 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.866999 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:23.867007 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.867007 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a70f3e-6112-4665-b57a-bbe22502fec3-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:23.867007 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.867014 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a70f3e-6112-4665-b57a-bbe22502fec3-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:23.867412 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.867023 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a70f3e-6112-4665-b57a-bbe22502fec3-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:23.990024 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:23.989991 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j"] Nov 02 07:47:24.001546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:24.001516 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-95s5j"] Nov 02 07:47:24.999763 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:24.999725 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:25.148770 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:25.148737 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:25.734229 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:25.734197 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:25.820484 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:25.820452 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a70f3e-6112-4665-b57a-bbe22502fec3" path="/var/lib/kubelet/pods/09a70f3e-6112-4665-b57a-bbe22502fec3/volumes" Nov 02 07:47:25.936589 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:25.936560 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:27.600243 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:27.600196 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:29.078889 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.078850 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx"] Nov 02 07:47:29.079385 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.079108 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" podUID="42a271e4-894f-44af-9786-ca603e3895e9" containerName="maas-api" containerID="cri-o://e0dcee5ac3fe6cca85493bb07d281eddba50a3345964975af466a8667f4fa8bf" gracePeriod=30 Nov 02 07:47:29.175116 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.175082 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:29.349844 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.349748 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:29.388225 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.388197 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:47:29.481255 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.481220 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:29.547451 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.547419 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-metrics-certs\") pod \"42a271e4-894f-44af-9786-ca603e3895e9\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " Nov 02 07:47:29.547658 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.547549 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a271e4-894f-44af-9786-ca603e3895e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a271e4-894f-44af-9786-ca603e3895e9-openshift-service-ca\") pod \"42a271e4-894f-44af-9786-ca603e3895e9\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " Nov 02 07:47:29.547658 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.547607 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-maas-api-tls\") pod \"42a271e4-894f-44af-9786-ca603e3895e9\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " Nov 02 07:47:29.547779 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.547670 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a271e4-894f-44af-9786-ca603e3895e9-kube-api-access-zv22d\" (UniqueName: \"kubernetes.io/projected/42a271e4-894f-44af-9786-ca603e3895e9-kube-api-access-zv22d\") pod \"42a271e4-894f-44af-9786-ca603e3895e9\" (UID: \"42a271e4-894f-44af-9786-ca603e3895e9\") " Nov 02 07:47:29.547982 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.547955 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a271e4-894f-44af-9786-ca603e3895e9-openshift-service-ca" pod "42a271e4-894f-44af-9786-ca603e3895e9" (UID: "42a271e4-894f-44af-9786-ca603e3895e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:47:29.548322 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.548299 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a271e4-894f-44af-9786-ca603e3895e9-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:29.550011 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.549984 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-metrics-certs" pod "42a271e4-894f-44af-9786-ca603e3895e9" (UID: "42a271e4-894f-44af-9786-ca603e3895e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:47:29.550114 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.550046 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a271e4-894f-44af-9786-ca603e3895e9-kube-api-access-zv22d" pod "42a271e4-894f-44af-9786-ca603e3895e9" (UID: "42a271e4-894f-44af-9786-ca603e3895e9"). InnerVolumeSpecName "kube-api-access-zv22d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:47:29.550221 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.550147 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-maas-api-tls" pod "42a271e4-894f-44af-9786-ca603e3895e9" (UID: "42a271e4-894f-44af-9786-ca603e3895e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:47:29.650451 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.650317 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:29.650451 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.650352 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv22d\" (UniqueName: \"kubernetes.io/projected/42a271e4-894f-44af-9786-ca603e3895e9-kube-api-access-zv22d\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:29.650451 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.650369 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a271e4-894f-44af-9786-ca603e3895e9-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:29.692253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.692208 5899 generic.go:356] "Generic (PLEG): container finished" podID="42a271e4-894f-44af-9786-ca603e3895e9" containerID="e0dcee5ac3fe6cca85493bb07d281eddba50a3345964975af466a8667f4fa8bf" exitCode=0 Nov 02 07:47:29.692455 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.692298 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" Nov 02 07:47:29.692455 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.692320 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" event={"ID":"42a271e4-894f-44af-9786-ca603e3895e9","Type":"ContainerDied","Data":"e0dcee5ac3fe6cca85493bb07d281eddba50a3345964975af466a8667f4fa8bf"} Nov 02 07:47:29.692455 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.692377 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx" event={"ID":"42a271e4-894f-44af-9786-ca603e3895e9","Type":"ContainerDied","Data":"e92b2322b7f35efac778e278f93f9e18f590b390eb44da2399327f013ea62f7e"} Nov 02 07:47:29.692455 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.692399 5899 scope.go:122] "RemoveContainer" containerID="e0dcee5ac3fe6cca85493bb07d281eddba50a3345964975af466a8667f4fa8bf" Nov 02 07:47:29.707226 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.707200 5899 scope.go:122] "RemoveContainer" containerID="e0dcee5ac3fe6cca85493bb07d281eddba50a3345964975af466a8667f4fa8bf" Nov 02 07:47:29.707604 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:47:29.707580 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0dcee5ac3fe6cca85493bb07d281eddba50a3345964975af466a8667f4fa8bf\": container with ID starting with e0dcee5ac3fe6cca85493bb07d281eddba50a3345964975af466a8667f4fa8bf not found: ID does not exist" containerID="e0dcee5ac3fe6cca85493bb07d281eddba50a3345964975af466a8667f4fa8bf" Nov 02 07:47:29.707710 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.707668 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0dcee5ac3fe6cca85493bb07d281eddba50a3345964975af466a8667f4fa8bf"} err="failed to get container status \"e0dcee5ac3fe6cca85493bb07d281eddba50a3345964975af466a8667f4fa8bf\": rpc error: code = NotFound desc = could not find container \"e0dcee5ac3fe6cca85493bb07d281eddba50a3345964975af466a8667f4fa8bf\": container with ID starting with e0dcee5ac3fe6cca85493bb07d281eddba50a3345964975af466a8667f4fa8bf not found: ID does not exist" Nov 02 07:47:29.745620 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.745585 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx"] Nov 02 07:47:29.748815 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.748790 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2cc1f-658cbddbf7-vpxwx"] Nov 02 07:47:29.827935 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:29.827900 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a271e4-894f-44af-9786-ca603e3895e9" path="/var/lib/kubelet/pods/42a271e4-894f-44af-9786-ca603e3895e9/volumes" Nov 02 07:47:30.035760 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:30.035710 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:31.689386 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:31.689351 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:31.883619 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:31.883585 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:32.479008 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:32.478971 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:32.849445 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:32.849352 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:33.701773 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:33.701733 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:34.361906 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:34.361871 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:34.844837 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:34.844807 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:34.998652 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:34.998587 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:35.850591 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:35.850551 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:38.366499 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:38.366467 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:38.575436 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:38.575400 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:38.732452 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:38.732420 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:39.472394 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:39.471723 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:40.056135 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:40.056098 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:40.251760 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:40.251718 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:40.392337 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:40.392249 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:40.547152 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:40.547121 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:40.966075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:40.966044 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:41.111005 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:41.110966 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:41.250847 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:41.250812 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:43.141761 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:47:43.141730 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:47:43.380410 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:43.375816 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:44.696380 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:44.696330 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:44.853792 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:44.853755 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:45.544174 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:45.544124 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:46.431031 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:46.430995 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:46.572501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:46.572470 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:47.664272 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:47.664235 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:47.870854 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:47.870821 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:48.011448 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:48.011403 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:48.461321 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:48.461218 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:48.668855 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:48.668822 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:48.928841 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:48.928797 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:49.081700 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:49.081661 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:49.821670 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:49.821633 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:49.970875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:49.970839 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:51.898617 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:51.898580 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:52.680750 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:52.680712 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:52.819545 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:52.819513 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:53.822252 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:53.822216 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:54.487977 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:54.487947 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:54.627286 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:54.627246 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:55.637016 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:55.636981 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6"] Nov 02 07:47:55.637501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:55.637243 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" podUID="81cf5af6-6278-476a-b0ec-d3c91e1693ee" containerName="maas-api" containerID="cri-o://65858e5762dc655308f8caadd28f996e45bc40006dab909f336a123e2fa0dcff" gracePeriod=30 Nov 02 07:47:55.732093 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:55.732036 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:55.909531 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:55.909251 5899 generic.go:356] "Generic (PLEG): container finished" podID="81cf5af6-6278-476a-b0ec-d3c91e1693ee" containerID="65858e5762dc655308f8caadd28f996e45bc40006dab909f336a123e2fa0dcff" exitCode=0 Nov 02 07:47:55.909531 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:55.909417 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" event={"ID":"81cf5af6-6278-476a-b0ec-d3c91e1693ee","Type":"ContainerDied","Data":"65858e5762dc655308f8caadd28f996e45bc40006dab909f336a123e2fa0dcff"} Nov 02 07:47:55.935097 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:55.934937 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:55.981132 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:55.981108 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:47:56.068517 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.068485 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:56.081309 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.081283 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81cf5af6-6278-476a-b0ec-d3c91e1693ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cf5af6-6278-476a-b0ec-d3c91e1693ee-openshift-service-ca\") pod \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " Nov 02 07:47:56.081486 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.081379 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-metrics-certs\") pod \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " Nov 02 07:47:56.081486 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.081401 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81cf5af6-6278-476a-b0ec-d3c91e1693ee-kube-api-access-gf8hm\" (UniqueName: \"kubernetes.io/projected/81cf5af6-6278-476a-b0ec-d3c91e1693ee-kube-api-access-gf8hm\") pod \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " Nov 02 07:47:56.081486 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.081457 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-maas-api-tls\") pod \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\" (UID: \"81cf5af6-6278-476a-b0ec-d3c91e1693ee\") " Nov 02 07:47:56.081882 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.081686 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cf5af6-6278-476a-b0ec-d3c91e1693ee-openshift-service-ca" pod "81cf5af6-6278-476a-b0ec-d3c91e1693ee" (UID: "81cf5af6-6278-476a-b0ec-d3c91e1693ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:47:56.081958 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.081920 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cf5af6-6278-476a-b0ec-d3c91e1693ee-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:56.083601 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.083578 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-metrics-certs" pod "81cf5af6-6278-476a-b0ec-d3c91e1693ee" (UID: "81cf5af6-6278-476a-b0ec-d3c91e1693ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:47:56.083673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.083641 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cf5af6-6278-476a-b0ec-d3c91e1693ee-kube-api-access-gf8hm" pod "81cf5af6-6278-476a-b0ec-d3c91e1693ee" (UID: "81cf5af6-6278-476a-b0ec-d3c91e1693ee"). InnerVolumeSpecName "kube-api-access-gf8hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:47:56.083673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.083648 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-maas-api-tls" pod "81cf5af6-6278-476a-b0ec-d3c91e1693ee" (UID: "81cf5af6-6278-476a-b0ec-d3c91e1693ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:47:56.183534 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.183456 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:56.183534 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.183492 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf8hm\" (UniqueName: \"kubernetes.io/projected/81cf5af6-6278-476a-b0ec-d3c91e1693ee-kube-api-access-gf8hm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:56.183534 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.183502 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81cf5af6-6278-476a-b0ec-d3c91e1693ee-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:47:56.659508 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.659427 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:56.919884 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.919804 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" Nov 02 07:47:56.919884 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.919838 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6" event={"ID":"81cf5af6-6278-476a-b0ec-d3c91e1693ee","Type":"ContainerDied","Data":"b1105904de0a4084962d354572c4f94f752da6f4107b4198aa6bbe3da7a5acec"} Nov 02 07:47:56.920110 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.919902 5899 scope.go:122] "RemoveContainer" containerID="65858e5762dc655308f8caadd28f996e45bc40006dab909f336a123e2fa0dcff" Nov 02 07:47:56.933896 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.933811 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:56.978041 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.977999 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6"] Nov 02 07:47:56.989655 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:56.989626 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc62c2-65bc78654d-z9gz6"] Nov 02 07:47:57.073142 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:57.073111 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:57.210589 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:57.210510 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:47:57.821578 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:57.821545 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81cf5af6-6278-476a-b0ec-d3c91e1693ee" path="/var/lib/kubelet/pods/81cf5af6-6278-476a-b0ec-d3c91e1693ee/volumes" Nov 02 07:47:59.004898 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:47:59.004849 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:00.133061 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:00.133022 5899 scope.go:122] "RemoveContainer" containerID="b8e03a29bc20578176d2f14e72cbec0c9dabd3f08f9c3452c3646b9323ec9286" Nov 02 07:48:00.704041 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:00.704006 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:02.517912 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:02.517878 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:02.660284 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:02.659031 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:02.854743 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:02.854657 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:02.991472 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:02.991439 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:03.130467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:03.130380 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:03.272793 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:03.272763 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:03.676335 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:03.676299 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:04.038126 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:04.038080 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:04.177369 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:04.177338 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:04.966318 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:04.966280 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:06.652920 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:06.652885 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:06.932929 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:06.932849 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:07.129970 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:07.129928 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:07.320071 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:07.319996 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:09.067452 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:09.067393 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:12.929064 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:12.929032 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:13.078988 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:13.078771 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:13.143730 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:48:13.143696 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:48:13.221882 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:13.221793 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:13.366108 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:13.366066 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:15.318879 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:15.318848 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:17.110432 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:17.110399 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:18.833383 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:18.833350 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:20.897122 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:20.897084 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:22.872397 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:22.872354 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:24.212148 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:24.212120 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:24.359556 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:24.359523 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:24.503728 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:24.503692 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:25.403040 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:25.403012 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:25.553217 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:25.553148 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:25.883622 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:25.883586 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:26.027568 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:26.027535 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:27.391325 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:27.391292 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:29.840447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:29.840404 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:30.218831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:30.218752 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:30.520973 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:30.520928 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:30.661485 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:30.661454 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:32.474337 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:32.474298 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:32.617593 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:32.617555 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:33.858387 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:33.858352 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:34.664006 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:34.663971 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:34.806206 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:34.805984 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:34.940417 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:34.940342 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:36.740179 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:36.740134 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:38.624768 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:38.624719 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:39.061224 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:39.061190 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:39.201664 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:39.201631 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:40.274823 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:40.274776 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:40.753749 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:40.753713 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:40.956576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:40.956544 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:41.464808 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:41.464738 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:42.131267 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:42.131210 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:43.162813 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:48:43.162775 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:48:43.764904 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:43.764861 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:44.280317 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:44.280284 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:45.644936 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:45.644906 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:45.785262 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:45.785223 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:47.173186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:47.173084 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:47.402089 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:47.402055 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:47.536090 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:47.536041 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:47.732819 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:47.732787 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:47.864926 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:47.864846 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:48.421685 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:48.421653 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:48.801642 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:48.801609 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:49.649491 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:49.649454 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:49.787882 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:49.787844 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:52.007144 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:52.007099 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:53.805246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:53.805212 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:55.428696 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:55.428666 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:55.576428 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:55.576392 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:55.709717 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:55.709630 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:55.852267 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:55.852219 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:56.639954 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:56.639919 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:57.702285 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:57.702244 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:48:59.837543 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:48:59.837468 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:01.629530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:01.629491 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:02.732232 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:02.732197 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:03.301090 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:03.301054 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:03.568134 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:03.568035 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:05.565232 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:05.565197 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:06.768816 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:06.768776 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:06.922755 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:06.922722 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:07.056939 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:07.056851 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:08.012879 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:08.012842 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:08.183435 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:08.183376 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:08.898761 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:08.898715 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:09.057188 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:09.055681 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:09.861589 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:09.861556 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:10.003502 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:10.003455 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:10.146321 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:10.146240 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:12.014687 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:12.014585 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:13.122756 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:13.122707 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:13.201890 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:49:13.201853 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:49:13.266829 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:13.266797 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:14.486749 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:14.486708 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:14.994285 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:14.994256 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:15.137521 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:15.137480 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:16.738149 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:16.738117 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:18.858087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:18.858050 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:19.657527 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:19.657496 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:20.602403 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:20.602372 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:21.167864 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:21.167827 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:21.356867 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:21.356771 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:21.581156 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:21.581114 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:23.506893 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:23.506860 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:25.268436 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:25.268403 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:25.415450 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:25.415420 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:25.563685 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:25.563582 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:26.272320 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:26.272283 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:26.988442 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:26.988410 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:27.133056 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:27.133020 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:27.325541 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:27.325416 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:27.636703 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:27.636618 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:27.777476 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:27.777430 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:28.113781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:28.113752 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:28.331052 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:28.330977 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:28.705766 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:28.705733 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:29.404253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:29.404221 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:31.373019 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:31.372987 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:33.582634 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:33.582601 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:33.992723 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:33.992688 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:34.317266 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:34.317189 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:34.450451 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:34.450355 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:34.582109 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:34.582029 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:35.513206 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:35.513178 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:37.635302 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:37.635214 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:37.768861 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:37.768831 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:37.904244 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:37.904142 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:39.423490 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:39.423456 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:40.078414 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:40.078371 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:40.509655 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:40.509610 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:41.586402 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:41.586351 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:41.725628 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:41.725579 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:42.470771 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:42.470718 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:43.141541 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:49:43.141511 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:49:44.743693 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:44.743658 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:45.935805 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:45.935777 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:46.088535 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:46.088497 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:46.587295 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:46.587260 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:46.778386 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:46.778355 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:47.162337 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:47.162304 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:47.928370 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:47.928340 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:48.611491 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:48.611455 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:49.251501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:49.251465 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:50.469443 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:50.469411 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:50.682771 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:50.682732 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:50.838848 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:50.838811 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:52.346845 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:52.346790 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:53.724335 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:53.724302 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:53.868625 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:53.868547 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:54.308444 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:54.308236 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:54.622705 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:54.622608 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:56.105514 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:56.105472 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:49:58.026568 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:49:58.026537 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:00.364563 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:00.364527 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:00.515556 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:00.515522 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:02.160603 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:02.160554 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:03.422728 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:03.422694 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:03.570237 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:03.568902 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:03.700942 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:03.700748 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:04.711771 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:04.711720 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:06.619822 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:06.619785 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:08.695493 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:08.695455 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:08.846717 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:08.846683 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:10.614513 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:10.614460 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:10.760512 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:10.760446 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:11.082494 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:11.082387 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:12.609893 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:12.609848 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:13.140808 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:50:13.140771 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:50:13.450585 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:13.450484 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:15.212953 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:15.212912 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:15.630055 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:15.630008 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:15.778599 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:15.778549 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:16.340698 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:16.340664 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:16.734028 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:16.733997 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:17.196972 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:17.196897 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:19.108489 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:19.108453 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:21.016019 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:21.015984 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:21.309926 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:21.309848 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:21.507473 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:21.507434 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:22.608806 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:22.608769 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:23.143095 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:23.143056 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:26.335200 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:26.335122 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:28.882335 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:28.882300 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:29.031745 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:29.031708 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:29.393269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:29.393237 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:29.539355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:29.539313 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:31.950660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:31.950627 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:34.003394 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:34.003345 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:35.537222 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:35.537184 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:35.792290 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:35.792201 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:35.949415 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:35.949382 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:36.523448 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:36.522873 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:38.456516 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:38.456480 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:40.381382 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:40.381348 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:41.657582 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:41.657546 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:41.837236 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:41.837197 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:42.986415 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:42.986378 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:43.150256 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:50:43.150218 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:50:44.453463 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:44.453431 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:44.597796 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:44.597767 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:44.734381 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:44.734349 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:45.223852 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:45.223767 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:45.600858 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:45.600826 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:46.463024 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:46.462988 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:46.852569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:46.852536 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:48.775365 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:48.775318 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:51.863736 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:51.863702 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:52.836092 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:52.836059 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:53.641927 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:53.641895 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:53.895748 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:53.895663 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:54.034925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:54.034892 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:54.597612 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:54.597574 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:56.364245 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:56.360585 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:58.338812 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:58.338726 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:59.750508 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:59.750471 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:50:59.890836 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:50:59.890800 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:00.221431 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:00.221338 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:00.589580 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:00.588946 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:00.816205 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:00.816155 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:01.007211 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:01.007154 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:02.523060 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:02.523021 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:02.670390 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:02.670355 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:03.172725 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:03.172697 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:03.345220 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:03.343969 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:03.638401 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:03.638367 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:06.164415 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:06.164367 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:51:06.916785 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:06.916743 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:07.109204 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:07.105177 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:08.868912 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:08.868871 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:09.032056 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:09.032023 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:09.444852 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:09.444806 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:11.335374 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:11.335336 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:13.141061 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:51:13.141024 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:51:13.285478 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:13.285445 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:15.811443 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:15.811409 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:16.347196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.347145 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg"] Nov 02 07:51:16.348285 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348256 5899 cpu_manager.go:414] "containerMap: removing container" podUID="3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d" containerName="maas-api" Nov 02 07:51:16.348450 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348428 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d" containerName="maas-api" Nov 02 07:51:16.348588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348474 5899 cpu_manager.go:414] "containerMap: removing container" podUID="09a70f3e-6112-4665-b57a-bbe22502fec3" containerName="istio-proxy" Nov 02 07:51:16.348588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348483 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a70f3e-6112-4665-b57a-bbe22502fec3" containerName="istio-proxy" Nov 02 07:51:16.348588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348506 5899 cpu_manager.go:414] "containerMap: removing container" podUID="42a271e4-894f-44af-9786-ca603e3895e9" containerName="maas-api" Nov 02 07:51:16.348588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348514 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a271e4-894f-44af-9786-ca603e3895e9" containerName="maas-api" Nov 02 07:51:16.348588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348528 5899 cpu_manager.go:414] "containerMap: removing container" podUID="b9c2b99e-5043-4611-ba0a-aa67c5b85e74" containerName="payload-pre-processing" Nov 02 07:51:16.348588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348535 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c2b99e-5043-4611-ba0a-aa67c5b85e74" containerName="payload-pre-processing" Nov 02 07:51:16.348588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348554 5899 cpu_manager.go:414] "containerMap: removing container" podUID="1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" containerName="istio-proxy" Nov 02 07:51:16.348588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348561 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" containerName="istio-proxy" Nov 02 07:51:16.348588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348580 5899 cpu_manager.go:414] "containerMap: removing container" podUID="81cf5af6-6278-476a-b0ec-d3c91e1693ee" containerName="maas-api" Nov 02 07:51:16.348588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348588 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cf5af6-6278-476a-b0ec-d3c91e1693ee" containerName="maas-api" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348602 5899 cpu_manager.go:414] "containerMap: removing container" podUID="e9ea184a-196b-4077-83fb-53c52fa8024f" containerName="payload-processing" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348609 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ea184a-196b-4077-83fb-53c52fa8024f" containerName="payload-processing" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348623 5899 cpu_manager.go:414] "containerMap: removing container" podUID="34975296-bf4a-4c2c-a4b2-0719760298da" containerName="istio-proxy" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348630 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34975296-bf4a-4c2c-a4b2-0719760298da" containerName="istio-proxy" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348654 5899 cpu_manager.go:414] "containerMap: removing container" podUID="64a6b79c-ecb0-43cd-b8b8-00e4922f01c2" containerName="maas-api" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348663 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a6b79c-ecb0-43cd-b8b8-00e4922f01c2" containerName="maas-api" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348818 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="34975296-bf4a-4c2c-a4b2-0719760298da" containerName="istio-proxy" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348832 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a6b79c-ecb0-43cd-b8b8-00e4922f01c2" containerName="maas-api" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348846 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4c94a1-e9c1-4c9b-a85e-d12a8c3d0a1d" containerName="maas-api" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348858 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a70f3e-6112-4665-b57a-bbe22502fec3" containerName="istio-proxy" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348870 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a271e4-894f-44af-9786-ca603e3895e9" containerName="maas-api" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348880 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c2b99e-5043-4611-ba0a-aa67c5b85e74" containerName="payload-pre-processing" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348890 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8c511f-15f3-4c8f-a3ad-ba854fd36fa8" containerName="istio-proxy" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348901 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cf5af6-6278-476a-b0ec-d3c91e1693ee" containerName="maas-api" Nov 02 07:51:16.349357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.348909 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ea184a-196b-4077-83fb-53c52fa8024f" containerName="payload-processing" Nov 02 07:51:16.352771 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.352750 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.387445 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.387384 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg"] Nov 02 07:51:16.474791 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.474742 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.475000 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.474892 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.475000 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.474934 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f03252-734a-44ad-9c2a-4072303b9a09-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.475124 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.475083 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.475200 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.475122 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.475200 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.475150 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.475271 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.475245 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.475377 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.475355 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-929jf\" (UniqueName: \"kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-kube-api-access-929jf\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.475425 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.475390 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.577001 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.576968 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.577225 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.577011 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.577225 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.577029 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.577225 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.577052 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.577225 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.577108 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-929jf\" (UniqueName: \"kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-kube-api-access-929jf\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.577225 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.577128 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.577225 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.577197 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.577564 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.577289 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.577564 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.577324 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f03252-734a-44ad-9c2a-4072303b9a09-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.577564 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.577524 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.577706 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.577597 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.577706 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.577606 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.577706 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.577680 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.578057 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.578039 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f03252-734a-44ad-9c2a-4072303b9a09-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.579722 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.579694 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.579828 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.579752 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.600338 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.600282 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-929jf\" (UniqueName: \"kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-kube-api-access-929jf\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.600338 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.600300 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.670918 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.670887 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:16.999305 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:16.999276 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg"] Nov 02 07:51:17.000572 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:51:17.000545 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f03252_734a_44ad_9c2a_4072303b9a09.slice/crio-307341106a3930e58c41f4f1a39a38e687194b42ac6999fe0281341b19103c04 WatchSource:0}: Error finding container 307341106a3930e58c41f4f1a39a38e687194b42ac6999fe0281341b19103c04: Status 404 returned error can't find the container with id 307341106a3930e58c41f4f1a39a38e687194b42ac6999fe0281341b19103c04 Nov 02 07:51:17.003290 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:17.003255 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:51:17.003406 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:17.003336 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:51:17.003406 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:17.003382 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:51:17.786644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:17.786609 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:17.986254 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:17.986216 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" event={"ID":"c1f03252-734a-44ad-9c2a-4072303b9a09","Type":"ContainerStarted","Data":"5f036726866260a4f0aa7f2c289ba6b46b09b5b5f37ae37f576aa786948e35d1"} Nov 02 07:51:17.986448 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:17.986279 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" event={"ID":"c1f03252-734a-44ad-9c2a-4072303b9a09","Type":"ContainerStarted","Data":"307341106a3930e58c41f4f1a39a38e687194b42ac6999fe0281341b19103c04"} Nov 02 07:51:18.082304 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:18.082191 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" podStartSLOduration=2.082151525 podStartE2EDuration="2.082151525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:18.050970624 +0000 UTC m=+1578.349251450" watchObservedRunningTime="2026-11-02 07:51:18.082151525 +0000 UTC m=+1578.380432382" Nov 02 07:51:18.680187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:18.676243 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:18.680187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:18.676400 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 07:51:18.680187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:18.676446 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" podUID="c1f03252-734a-44ad-9c2a-4072303b9a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 07:51:19.631725 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:19.631693 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:19.682465 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:19.675342 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 07:51:19.682465 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:19.675412 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" podUID="c1f03252-734a-44ad-9c2a-4072303b9a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 07:51:20.677230 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:20.677189 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 07:51:20.677810 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:20.677779 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" podUID="c1f03252-734a-44ad-9c2a-4072303b9a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 07:51:21.527481 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:21.527447 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:21.682263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:21.676598 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 07:51:21.682263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:21.676662 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" podUID="c1f03252-734a-44ad-9c2a-4072303b9a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 07:51:22.677651 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:22.677612 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 07:51:22.677934 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:22.677904 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" podUID="c1f03252-734a-44ad-9c2a-4072303b9a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 07:51:23.404733 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:23.404671 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:23.672368 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:23.672279 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 07:51:23.672368 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:23.672355 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" podUID="c1f03252-734a-44ad-9c2a-4072303b9a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 07:51:24.676848 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:24.676816 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:24.677255 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:24.677087 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:24.677811 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:24.677790 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:51:25.388534 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:25.388499 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:27.651095 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:27.651058 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:27.791565 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:27.791528 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:28.093556 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:28.093520 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:30.805117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:30.805081 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:30.896009 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:30.895961 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:51:31.084622 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:31.084449 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:31.478113 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:31.478077 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:32.608527 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:32.608487 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s"] Nov 02 07:51:32.608935 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:32.608704 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" podUID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" containerName="istio-proxy" containerID="cri-o://80e640484c1d52411e4a2e0451d2f2abf2738587b4a25ec5068981fa25446eb4" gracePeriod=30 Nov 02 07:51:33.682293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:33.682254 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:34.088815 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:34.088779 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:35.428697 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:35.428664 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:37.218457 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.218417 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:37.881307 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.881245 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:51:37.996047 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.996009 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-socket\") pod \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " Nov 02 07:51:37.996254 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.996106 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-envoy\") pod \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " Nov 02 07:51:37.996254 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.996190 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-kube-api-access-g2spp\" (UniqueName: \"kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-kube-api-access-g2spp\") pod \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " Nov 02 07:51:37.996254 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.996238 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-certs\") pod \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " Nov 02 07:51:37.996442 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.996291 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-credential-socket\") pod \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " Nov 02 07:51:37.996442 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.996324 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istiod-ca-cert\") pod \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " Nov 02 07:51:37.996442 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.996369 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-socket" pod "1ececa38-98c8-4d62-b4fc-1f06ab4e887c" (UID: "1ececa38-98c8-4d62-b4fc-1f06ab4e887c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:51:37.996597 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.996464 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-podinfo\") pod \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " Nov 02 07:51:37.996597 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.996583 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-data\") pod \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " Nov 02 07:51:37.996690 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.996587 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-certs" pod "1ececa38-98c8-4d62-b4fc-1f06ab4e887c" (UID: "1ececa38-98c8-4d62-b4fc-1f06ab4e887c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:51:37.996690 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.996617 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-token\" (UniqueName: \"kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-token\") pod \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\" (UID: \"1ececa38-98c8-4d62-b4fc-1f06ab4e887c\") " Nov 02 07:51:37.996690 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.996628 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-credential-socket" pod "1ececa38-98c8-4d62-b4fc-1f06ab4e887c" (UID: "1ececa38-98c8-4d62-b4fc-1f06ab4e887c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:51:37.996853 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.996793 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-data" pod "1ececa38-98c8-4d62-b4fc-1f06ab4e887c" (UID: "1ececa38-98c8-4d62-b4fc-1f06ab4e887c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:51:37.997177 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.997134 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istiod-ca-cert" pod "1ececa38-98c8-4d62-b4fc-1f06ab4e887c" (UID: "1ececa38-98c8-4d62-b4fc-1f06ab4e887c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:51:37.997359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.997330 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:51:37.997438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.997357 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:51:37.997438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.997373 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:51:37.997438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.997387 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:51:37.997438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.997401 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:51:37.998508 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.998487 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-envoy" pod "1ececa38-98c8-4d62-b4fc-1f06ab4e887c" (UID: "1ececa38-98c8-4d62-b4fc-1f06ab4e887c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:51:37.998618 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.998541 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-kube-api-access-g2spp" pod "1ececa38-98c8-4d62-b4fc-1f06ab4e887c" (UID: "1ececa38-98c8-4d62-b4fc-1f06ab4e887c"). InnerVolumeSpecName "kube-api-access-g2spp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:51:37.998687 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.998622 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-podinfo" pod "1ececa38-98c8-4d62-b4fc-1f06ab4e887c" (UID: "1ececa38-98c8-4d62-b4fc-1f06ab4e887c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:51:37.998774 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:37.998760 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-token" pod "1ececa38-98c8-4d62-b4fc-1f06ab4e887c" (UID: "1ececa38-98c8-4d62-b4fc-1f06ab4e887c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:51:38.099236 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:38.099194 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:51:38.099236 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:38.099229 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:51:38.099236 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:38.099240 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:51:38.099464 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:38.099252 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2spp\" (UniqueName: \"kubernetes.io/projected/1ececa38-98c8-4d62-b4fc-1f06ab4e887c-kube-api-access-g2spp\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:51:38.282279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:38.282240 5899 generic.go:356] "Generic (PLEG): container finished" podID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" containerID="80e640484c1d52411e4a2e0451d2f2abf2738587b4a25ec5068981fa25446eb4" exitCode=0 Nov 02 07:51:38.282708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:38.282284 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" Nov 02 07:51:38.282708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:38.282326 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" event={"ID":"1ececa38-98c8-4d62-b4fc-1f06ab4e887c","Type":"ContainerDied","Data":"80e640484c1d52411e4a2e0451d2f2abf2738587b4a25ec5068981fa25446eb4"} Nov 02 07:51:38.282708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:38.282384 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s" event={"ID":"1ececa38-98c8-4d62-b4fc-1f06ab4e887c","Type":"ContainerDied","Data":"fd847513fb465c2465681c036534778d4ed51f3eb178f5ae45346e6c9a335e8d"} Nov 02 07:51:38.282708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:38.282411 5899 scope.go:122] "RemoveContainer" containerID="80e640484c1d52411e4a2e0451d2f2abf2738587b4a25ec5068981fa25446eb4" Nov 02 07:51:38.293799 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:38.293777 5899 scope.go:122] "RemoveContainer" containerID="80e640484c1d52411e4a2e0451d2f2abf2738587b4a25ec5068981fa25446eb4" Nov 02 07:51:38.294269 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:51:38.294145 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e640484c1d52411e4a2e0451d2f2abf2738587b4a25ec5068981fa25446eb4\": container with ID starting with 80e640484c1d52411e4a2e0451d2f2abf2738587b4a25ec5068981fa25446eb4 not found: ID does not exist" containerID="80e640484c1d52411e4a2e0451d2f2abf2738587b4a25ec5068981fa25446eb4" Nov 02 07:51:38.294360 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:38.294281 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e640484c1d52411e4a2e0451d2f2abf2738587b4a25ec5068981fa25446eb4"} err="failed to get container status \"80e640484c1d52411e4a2e0451d2f2abf2738587b4a25ec5068981fa25446eb4\": rpc error: code = NotFound desc = could not find container \"80e640484c1d52411e4a2e0451d2f2abf2738587b4a25ec5068981fa25446eb4\": container with ID starting with 80e640484c1d52411e4a2e0451d2f2abf2738587b4a25ec5068981fa25446eb4 not found: ID does not exist" Nov 02 07:51:38.333977 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:38.333940 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s"] Nov 02 07:51:38.345739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:38.345706 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-mqr2s"] Nov 02 07:51:39.126484 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:39.126451 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:39.581497 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:39.581467 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:39.820563 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:39.820523 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" path="/var/lib/kubelet/pods/1ececa38-98c8-4d62-b4fc-1f06ab4e887c/volumes" Nov 02 07:51:41.405943 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:41.405909 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:41.823479 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:41.823449 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:41.961745 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:41.961707 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:43.141710 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:51:43.141675 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:51:43.350180 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:43.341657 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:46.001617 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:46.001577 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:47.925985 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:47.925951 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:49.743204 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:49.743146 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:52.031992 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:52.031944 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:53.767332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:53.767271 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:55.606988 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:55.606953 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:57.465155 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:57.465120 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:57.605240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:57.605185 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:58.592767 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:58.592730 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:59.058023 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:59.057974 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt"] Nov 02 07:51:59.058587 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:59.058520 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" podUID="88387240-8d67-4c5a-b8b2-940c0f6d2d0c" containerName="istio-proxy" containerID="cri-o://30b2629bc957ecd6a3007046cfb8397b5bb340e76c9b312a7d3384f9ca424afa" gracePeriod=30 Nov 02 07:51:59.121470 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:59.121435 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:59.276315 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:59.276277 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:51:59.421848 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:51:59.421697 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:01.702231 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:01.702191 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:03.082236 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:03.082155 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:03.229459 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:03.229425 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:03.641336 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:03.641297 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:03.770946 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:03.770914 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:04.330154 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.330085 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:52:04.458526 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.458487 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-podinfo\") pod \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " Nov 02 07:52:04.458732 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.458565 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istiod-ca-cert\") pod \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " Nov 02 07:52:04.458732 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.458637 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-socket\") pod \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " Nov 02 07:52:04.458732 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.458684 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-credential-socket\") pod \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " Nov 02 07:52:04.458732 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.458728 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-token\" (UniqueName: \"kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-token\") pod \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " Nov 02 07:52:04.458957 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.458750 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-data\") pod \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " Nov 02 07:52:04.458957 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.458813 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-kube-api-access-ljv4n\" (UniqueName: \"kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-kube-api-access-ljv4n\") pod \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " Nov 02 07:52:04.458957 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.458940 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-certs\") pod \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " Nov 02 07:52:04.458957 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.458917 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-socket" pod "88387240-8d67-4c5a-b8b2-940c0f6d2d0c" (UID: "88387240-8d67-4c5a-b8b2-940c0f6d2d0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:52:04.459182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.458982 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-envoy\") pod \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\" (UID: \"88387240-8d67-4c5a-b8b2-940c0f6d2d0c\") " Nov 02 07:52:04.459182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.458985 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-credential-socket" pod "88387240-8d67-4c5a-b8b2-940c0f6d2d0c" (UID: "88387240-8d67-4c5a-b8b2-940c0f6d2d0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:52:04.459182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.459003 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istiod-ca-cert" pod "88387240-8d67-4c5a-b8b2-940c0f6d2d0c" (UID: "88387240-8d67-4c5a-b8b2-940c0f6d2d0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:52:04.459182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.459156 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-data" pod "88387240-8d67-4c5a-b8b2-940c0f6d2d0c" (UID: "88387240-8d67-4c5a-b8b2-940c0f6d2d0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:52:04.459355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.459206 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-certs" pod "88387240-8d67-4c5a-b8b2-940c0f6d2d0c" (UID: "88387240-8d67-4c5a-b8b2-940c0f6d2d0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:52:04.459641 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.459617 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:52:04.459641 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.459643 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:52:04.459842 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.459657 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:52:04.459842 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.459670 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:52:04.459842 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.459684 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:52:04.460868 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.460844 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-podinfo" pod "88387240-8d67-4c5a-b8b2-940c0f6d2d0c" (UID: "88387240-8d67-4c5a-b8b2-940c0f6d2d0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:52:04.460970 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.460943 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-envoy" pod "88387240-8d67-4c5a-b8b2-940c0f6d2d0c" (UID: "88387240-8d67-4c5a-b8b2-940c0f6d2d0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:52:04.461060 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.461041 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-token" pod "88387240-8d67-4c5a-b8b2-940c0f6d2d0c" (UID: "88387240-8d67-4c5a-b8b2-940c0f6d2d0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:52:04.461132 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.461119 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-kube-api-access-ljv4n" pod "88387240-8d67-4c5a-b8b2-940c0f6d2d0c" (UID: "88387240-8d67-4c5a-b8b2-940c0f6d2d0c"). InnerVolumeSpecName "kube-api-access-ljv4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:52:04.519607 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.519567 5899 generic.go:356] "Generic (PLEG): container finished" podID="88387240-8d67-4c5a-b8b2-940c0f6d2d0c" containerID="30b2629bc957ecd6a3007046cfb8397b5bb340e76c9b312a7d3384f9ca424afa" exitCode=0 Nov 02 07:52:04.519812 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.519620 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" Nov 02 07:52:04.519812 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.519653 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" event={"ID":"88387240-8d67-4c5a-b8b2-940c0f6d2d0c","Type":"ContainerDied","Data":"30b2629bc957ecd6a3007046cfb8397b5bb340e76c9b312a7d3384f9ca424afa"} Nov 02 07:52:04.519812 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.519722 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt" event={"ID":"88387240-8d67-4c5a-b8b2-940c0f6d2d0c","Type":"ContainerDied","Data":"efc25db8b186c5412e38ca710b5aeeb499efee219549338e36cc6c85ceea13f9"} Nov 02 07:52:04.519812 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.519753 5899 scope.go:122] "RemoveContainer" containerID="30b2629bc957ecd6a3007046cfb8397b5bb340e76c9b312a7d3384f9ca424afa" Nov 02 07:52:04.530879 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.530862 5899 scope.go:122] "RemoveContainer" containerID="30b2629bc957ecd6a3007046cfb8397b5bb340e76c9b312a7d3384f9ca424afa" Nov 02 07:52:04.531199 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:52:04.531139 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b2629bc957ecd6a3007046cfb8397b5bb340e76c9b312a7d3384f9ca424afa\": container with ID starting with 30b2629bc957ecd6a3007046cfb8397b5bb340e76c9b312a7d3384f9ca424afa not found: ID does not exist" containerID="30b2629bc957ecd6a3007046cfb8397b5bb340e76c9b312a7d3384f9ca424afa" Nov 02 07:52:04.531291 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.531216 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b2629bc957ecd6a3007046cfb8397b5bb340e76c9b312a7d3384f9ca424afa"} err="failed to get container status \"30b2629bc957ecd6a3007046cfb8397b5bb340e76c9b312a7d3384f9ca424afa\": rpc error: code = NotFound desc = could not find container \"30b2629bc957ecd6a3007046cfb8397b5bb340e76c9b312a7d3384f9ca424afa\": container with ID starting with 30b2629bc957ecd6a3007046cfb8397b5bb340e76c9b312a7d3384f9ca424afa not found: ID does not exist" Nov 02 07:52:04.561234 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.561204 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:52:04.561234 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.561229 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:52:04.561234 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.561238 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:52:04.561447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.561248 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljv4n\" (UniqueName: \"kubernetes.io/projected/88387240-8d67-4c5a-b8b2-940c0f6d2d0c-kube-api-access-ljv4n\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:52:04.574675 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.574283 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt"] Nov 02 07:52:04.581440 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.581355 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-29cwt"] Nov 02 07:52:04.775530 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.775498 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:04.966346 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:04.966255 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:05.607995 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:05.607954 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:05.819658 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:05.819625 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88387240-8d67-4c5a-b8b2-940c0f6d2d0c" path="/var/lib/kubelet/pods/88387240-8d67-4c5a-b8b2-940c0f6d2d0c/volumes" Nov 02 07:52:07.411361 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:07.411322 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:09.098926 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:09.098894 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:11.139825 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:11.139742 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:13.142261 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:52:13.142226 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:52:13.504109 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:13.504072 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:15.264951 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:15.264916 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:16.087729 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:16.087690 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:16.642272 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:16.642241 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:16.778325 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:16.778271 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:17.232580 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:17.232531 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:17.609893 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:17.609795 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:19.346770 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:19.346735 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:21.605427 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:21.605361 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:23.061554 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:23.061519 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:23.272627 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:23.272585 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:23.416005 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:23.415912 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:23.638082 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:23.638036 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:25.331055 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:25.331018 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:27.289368 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:27.289335 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:29.089308 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:29.089271 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:29.242919 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:29.242885 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:30.798132 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:30.798086 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:32.878459 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:32.878424 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:35.172461 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:35.171259 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:52:35.183932 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:35.183343 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" podUID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 07:52:35.401199 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:35.399310 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 07:52:37.234742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:37.234696 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:39.141921 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:39.141865 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:40.717472 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:40.717405 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:41.677323 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:41.677280 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:42.874503 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:42.874471 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:43.141032 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:52:43.140954 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:52:43.469746 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:43.469667 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:43.949441 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:43.949406 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:44.093502 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.093467 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:44.217588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.217498 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5"] Nov 02 07:52:44.218078 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.218062 5899 cpu_manager.go:414] "containerMap: removing container" podUID="88387240-8d67-4c5a-b8b2-940c0f6d2d0c" containerName="istio-proxy" Nov 02 07:52:44.218078 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.218078 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88387240-8d67-4c5a-b8b2-940c0f6d2d0c" containerName="istio-proxy" Nov 02 07:52:44.218244 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.218105 5899 cpu_manager.go:414] "containerMap: removing container" podUID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" containerName="istio-proxy" Nov 02 07:52:44.218244 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.218112 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" containerName="istio-proxy" Nov 02 07:52:44.218342 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.218287 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="88387240-8d67-4c5a-b8b2-940c0f6d2d0c" containerName="istio-proxy" Nov 02 07:52:44.218342 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.218301 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ececa38-98c8-4d62-b4fc-1f06ab4e887c" containerName="istio-proxy" Nov 02 07:52:44.221409 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.221390 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.260829 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.260785 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5"] Nov 02 07:52:44.269507 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.269454 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.269751 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.269558 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.269751 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.269623 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.269861 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.269778 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.269861 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.269816 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5882a478-2c3b-4ead-976c-cc4231e32e12-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.269953 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.269899 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.269953 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.269947 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.270053 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.270036 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czh5r\" (UniqueName: \"kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-kube-api-access-czh5r\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.270236 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.270201 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.331987 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.331951 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9ab4d2-openshift-default-dockercfg-xtcp6\"" Nov 02 07:52:44.372238 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.372198 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czh5r\" (UniqueName: \"kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-kube-api-access-czh5r\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.372440 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.372260 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.372440 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.372328 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.372440 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.372378 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.372440 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.372431 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.372655 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.372553 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.372655 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.372580 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5882a478-2c3b-4ead-976c-cc4231e32e12-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.372761 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.372665 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.372761 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.372713 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.372951 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.372924 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.373153 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.372998 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.373276 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.373131 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.373375 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.373350 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.373441 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.373423 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5882a478-2c3b-4ead-976c-cc4231e32e12-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.374949 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.374925 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.375088 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.375071 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.395139 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.395105 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czh5r\" (UniqueName: \"kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-kube-api-access-czh5r\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.395357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.395334 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.538844 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.538816 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:44.690508 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.690473 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:44.852802 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.852777 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5"] Nov 02 07:52:44.854718 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:52:44.854691 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5882a478_2c3b_4ead_976c_cc4231e32e12.slice/crio-1a9e844e0b45f00f269e0e3426016864ea8c832be825a4855aab969570cbbec2 WatchSource:0}: Error finding container 1a9e844e0b45f00f269e0e3426016864ea8c832be825a4855aab969570cbbec2: Status 404 returned error can't find the container with id 1a9e844e0b45f00f269e0e3426016864ea8c832be825a4855aab969570cbbec2 Nov 02 07:52:44.857379 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.857349 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:52:44.857463 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.857411 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:52:44.857463 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.857449 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:52:44.898552 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:44.898518 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" event={"ID":"5882a478-2c3b-4ead-976c-cc4231e32e12","Type":"ContainerStarted","Data":"1a9e844e0b45f00f269e0e3426016864ea8c832be825a4855aab969570cbbec2"} Nov 02 07:52:45.058363 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:45.058330 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:45.905992 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:45.905952 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" event={"ID":"5882a478-2c3b-4ead-976c-cc4231e32e12","Type":"ContainerStarted","Data":"976e2190a5f68d001b0914d8207e5386f1536f95bfeb671543b168aef40bddbe"} Nov 02 07:52:45.979184 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:45.978350 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" podStartSLOduration=1.9783299030000001 podStartE2EDuration="1.978329903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:45.974636733 +0000 UTC m=+1666.272917588" watchObservedRunningTime="2026-11-02 07:52:45.978329903 +0000 UTC m=+1666.276610729" Nov 02 07:52:46.281493 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:46.281437 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:46.539860 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:46.539776 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:46.541603 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:46.541579 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 07:52:46.541728 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:46.541624 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" podUID="5882a478-2c3b-4ead-976c-cc4231e32e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 07:52:47.539900 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:47.539863 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 07:52:47.540334 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:47.539919 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" podUID="5882a478-2c3b-4ead-976c-cc4231e32e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 07:52:47.909896 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:47.909814 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:48.478186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:48.468602 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:48.539840 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:48.539801 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 07:52:48.540066 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:48.539860 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" podUID="5882a478-2c3b-4ead-976c-cc4231e32e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 07:52:49.015603 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:49.015562 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:49.507614 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:49.507576 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:49.539946 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:49.539909 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 07:52:49.540102 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:49.539974 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" podUID="5882a478-2c3b-4ead-976c-cc4231e32e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 07:52:49.801243 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:49.800572 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:50.232738 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:50.232710 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:50.564237 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:50.551507 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 07:52:50.564237 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:50.551573 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" podUID="5882a478-2c3b-4ead-976c-cc4231e32e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 07:52:50.654154 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:50.654100 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:51.024886 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:51.024720 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:51.163490 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:51.163444 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:51.546900 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:51.546865 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:51.547401 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:51.547377 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:51.548221 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:51.548200 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:52:53.328669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:53.328639 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:53.668233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:53.668130 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:53.816691 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:53.816641 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:53.989548 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:53.989511 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:55.344912 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:55.344878 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:55.493876 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:55.493848 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:55.659384 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:55.659286 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:55.871766 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:55.871711 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:57.441020 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:57.440983 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:58.012113 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:58.012078 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:58.153217 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:58.153155 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:58.619706 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:58.619674 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:58.757503 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:58.757456 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:52:59.643803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:52:59.643764 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:00.905751 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:00.905718 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:01.049187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:01.046746 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:01.960505 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:01.960468 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:02.156183 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:02.152742 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:02.943534 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:02.943501 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:03.879728 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:03.879694 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:04.125349 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:04.125315 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:04.265847 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:04.265807 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:04.743031 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:04.742988 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:06.551891 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:06.551856 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:07.700078 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:07.700031 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:07.859980 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:07.859943 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:08.236560 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:08.236513 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:08.385155 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:08.385124 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:09.161057 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:09.161019 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:09.369736 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:09.369702 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:09.569538 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:09.569502 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:10.349285 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:10.349256 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:11.115431 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:11.115390 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:12.619648 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:12.619603 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:13.144707 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:53:13.144671 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:53:13.466491 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:13.466415 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:13.596960 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:13.596923 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:14.309075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.309041 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f"] Nov 02 07:53:14.314443 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.314418 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.344038 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.341804 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f"] Nov 02 07:53:14.429365 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.429328 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.429536 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.429486 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.429601 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.429553 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.429652 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.429622 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.429750 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.429735 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.429791 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.429768 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.429841 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.429790 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59tzt\" (UniqueName: \"kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-kube-api-access-59tzt\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.429886 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.429868 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.429931 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.429893 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.532196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.532147 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.532464 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.532444 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.532575 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.532491 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59tzt\" (UniqueName: \"kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-kube-api-access-59tzt\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.532575 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.532533 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.532681 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.532573 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.532681 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.532623 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.532779 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.532725 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.532779 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.532758 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.532882 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.532852 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.532938 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.532878 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.533671 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.533340 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.533671 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.533428 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.535063 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.534059 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.535931 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.535890 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.538098 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.536485 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.538527 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.538503 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.555473 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.555447 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.555653 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.555634 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59tzt\" (UniqueName: \"kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-kube-api-access-59tzt\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.633300 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.633205 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:14.961375 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:53:14.961338 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14a0685_a3b9_4b26_abe8_d5ac282cd653.slice/crio-5defe3d25201ef2b64b91bdcd66f36c0181a858bd5dccb392dd587b456b29b8a WatchSource:0}: Error finding container 5defe3d25201ef2b64b91bdcd66f36c0181a858bd5dccb392dd587b456b29b8a: Status 404 returned error can't find the container with id 5defe3d25201ef2b64b91bdcd66f36c0181a858bd5dccb392dd587b456b29b8a Nov 02 07:53:14.964270 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.964245 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f"] Nov 02 07:53:14.964895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.964865 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:53:14.964981 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.964926 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:53:14.973657 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:14.965412 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:53:15.124571 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:15.124530 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" event={"ID":"a14a0685-a3b9-4b26-abe8-d5ac282cd653","Type":"ContainerStarted","Data":"a62794f8f51434f8542ab507552bda721d872e0253c6f0174608066b9ad5f2ba"} Nov 02 07:53:15.124766 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:15.124634 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" event={"ID":"a14a0685-a3b9-4b26-abe8-d5ac282cd653","Type":"ContainerStarted","Data":"5defe3d25201ef2b64b91bdcd66f36c0181a858bd5dccb392dd587b456b29b8a"} Nov 02 07:53:15.169411 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:15.169356 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" podStartSLOduration=1.169339345 podStartE2EDuration="1.169339345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:15.16857028 +0000 UTC m=+1695.466851108" watchObservedRunningTime="2026-11-02 07:53:15.169339345 +0000 UTC m=+1695.467620174" Nov 02 07:53:15.500504 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:15.500198 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:15.633510 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:15.633476 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:16.318831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:16.318800 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:16.455418 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:16.455333 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:16.635437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:16.635356 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 07:53:16.635437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:16.635411 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" podUID="a14a0685-a3b9-4b26-abe8-d5ac282cd653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 07:53:17.182238 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:17.179960 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:17.633713 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:17.633676 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 07:53:17.633899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:17.633730 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" podUID="a14a0685-a3b9-4b26-abe8-d5ac282cd653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 07:53:17.635268 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:17.635243 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:17.771618 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:17.771583 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:17.903229 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:17.903114 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:18.634076 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:18.634043 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 07:53:18.634280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:18.634098 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" podUID="a14a0685-a3b9-4b26-abe8-d5ac282cd653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 07:53:19.324240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:19.324206 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:19.468713 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:19.468678 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:19.634509 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:19.634431 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 07:53:19.634509 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:19.634493 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" podUID="a14a0685-a3b9-4b26-abe8-d5ac282cd653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 07:53:19.889298 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:19.889220 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:20.283763 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:20.283725 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:20.633832 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:20.633748 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 07:53:20.633832 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:20.633812 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" podUID="a14a0685-a3b9-4b26-abe8-d5ac282cd653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 07:53:20.682214 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:20.680104 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:21.638254 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:21.638222 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:21.638666 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:21.638481 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:21.639256 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:21.639237 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:22.091057 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:22.091013 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:23.106125 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:23.106094 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:23.827465 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:23.827432 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:24.845942 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:24.845906 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:26.114955 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:26.114920 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:26.544031 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:26.543992 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:28.307341 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:28.307303 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:29.333969 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.333928 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp"] Nov 02 07:53:29.337711 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.337689 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.361079 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.361037 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp"] Nov 02 07:53:29.516113 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.516082 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.516284 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.516212 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.516349 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.516289 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxqp\" (UniqueName: \"kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-kube-api-access-zvxqp\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.516349 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.516336 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.516433 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.516364 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.516550 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.516530 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.516606 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.516588 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.516709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.516690 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95439f08-2283-44e9-a424-bb6f9e45fef0-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.516775 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.516733 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.617884 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.617794 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.617884 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.617856 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxqp\" (UniqueName: \"kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-kube-api-access-zvxqp\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.618126 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.617906 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.618126 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.617939 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.618126 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.618033 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.618126 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.618086 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.618419 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.618143 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95439f08-2283-44e9-a424-bb6f9e45fef0-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.618419 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.618210 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.618419 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.618272 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.618419 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.618362 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.618419 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.618389 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.618694 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.618561 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.618739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.618702 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.618921 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.618903 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95439f08-2283-44e9-a424-bb6f9e45fef0-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.620404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.620389 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.620625 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.620606 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.641108 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.641084 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.641363 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.641346 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxqp\" (UniqueName: \"kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-kube-api-access-zvxqp\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.653264 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.653241 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:29.972724 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.972693 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp"] Nov 02 07:53:29.973982 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:53:29.973954 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95439f08_2283_44e9_a424_bb6f9e45fef0.slice/crio-ed507e6e94f4f07b6f7248c6344fc8f066b021b9f3a5a817be49aeaa4923ff20 WatchSource:0}: Error finding container ed507e6e94f4f07b6f7248c6344fc8f066b021b9f3a5a817be49aeaa4923ff20: Status 404 returned error can't find the container with id ed507e6e94f4f07b6f7248c6344fc8f066b021b9f3a5a817be49aeaa4923ff20 Nov 02 07:53:29.976537 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.976509 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:53:29.976603 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.976576 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:53:29.976647 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:29.976605 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:53:30.106256 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:30.106224 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:30.243361 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:30.243324 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" event={"ID":"95439f08-2283-44e9-a424-bb6f9e45fef0","Type":"ContainerStarted","Data":"72c1e62303e1e2f06f9100a9745401fe60b8c8083254fea0915e816f29a37636"} Nov 02 07:53:30.243520 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:30.243382 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" event={"ID":"95439f08-2283-44e9-a424-bb6f9e45fef0","Type":"ContainerStarted","Data":"ed507e6e94f4f07b6f7248c6344fc8f066b021b9f3a5a817be49aeaa4923ff20"} Nov 02 07:53:30.289360 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:30.289307 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" podStartSLOduration=1.289291139 podStartE2EDuration="1.289291139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:30.287915606 +0000 UTC m=+1710.586196432" watchObservedRunningTime="2026-11-02 07:53:30.289291139 +0000 UTC m=+1710.587571964" Nov 02 07:53:30.653825 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:30.653747 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:31.587907 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:31.587862 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:31.655243 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:31.655203 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 07:53:31.655828 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:31.655791 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" podUID="95439f08-2283-44e9-a424-bb6f9e45fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 07:53:32.436832 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:32.436797 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:32.578041 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:32.577983 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:32.653687 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:32.653655 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 07:53:32.653857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:32.653724 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" podUID="95439f08-2283-44e9-a424-bb6f9e45fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 07:53:33.654457 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:33.654422 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 07:53:33.654859 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:33.654475 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" podUID="95439f08-2283-44e9-a424-bb6f9e45fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 07:53:34.442036 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:34.442000 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:34.654113 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:34.654075 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 07:53:34.654348 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:34.654130 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" podUID="95439f08-2283-44e9-a424-bb6f9e45fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 07:53:35.653810 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:35.653770 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 07:53:35.654207 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:35.653832 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" podUID="95439f08-2283-44e9-a424-bb6f9e45fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 07:53:36.419795 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:36.419759 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:36.653716 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:36.653682 5899 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 07:53:36.653878 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:36.653745 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" podUID="95439f08-2283-44e9-a424-bb6f9e45fef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 07:53:37.658594 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:37.658517 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:37.659055 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:37.658963 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:37.659653 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:37.659632 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:38.284576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:38.284541 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:38.955803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:38.955767 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:39.107306 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:39.107271 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:39.245742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:39.245714 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:39.374757 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:39.374721 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:39.686327 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:39.686252 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:39.850298 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:39.850265 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:40.009248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:40.009212 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:40.177950 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:40.177919 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5"] Nov 02 07:53:40.178202 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:40.178174 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" podUID="5882a478-2c3b-4ead-976c-cc4231e32e12" containerName="istio-proxy" containerID="cri-o://976e2190a5f68d001b0914d8207e5386f1536f95bfeb671543b168aef40bddbe" gracePeriod=30 Nov 02 07:53:40.182477 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:40.182399 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:40.187830 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:40.187803 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f"] Nov 02 07:53:40.188190 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:40.188136 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" podUID="a14a0685-a3b9-4b26-abe8-d5ac282cd653" containerName="istio-proxy" containerID="cri-o://a62794f8f51434f8542ab507552bda721d872e0253c6f0174608066b9ad5f2ba" gracePeriod=30 Nov 02 07:53:40.197098 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:40.197050 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp"] Nov 02 07:53:40.334730 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:40.334550 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:40.393183 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:40.390979 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" podUID="95439f08-2283-44e9-a424-bb6f9e45fef0" containerName="istio-proxy" containerID="cri-o://72c1e62303e1e2f06f9100a9745401fe60b8c8083254fea0915e816f29a37636" gracePeriod=30 Nov 02 07:53:40.484616 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:40.484582 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:40.751958 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:40.751918 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:40.978592 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:40.978556 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:41.124868 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:41.124784 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:41.338428 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:41.338395 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:41.817540 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:41.817504 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:41.976023 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:41.975199 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:42.639068 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:42.639037 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:43.141839 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:53:43.141801 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:53:44.163753 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:44.163715 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:45.523176 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.522590 5899 generic.go:356] "Generic (PLEG): container finished" podID="a14a0685-a3b9-4b26-abe8-d5ac282cd653" containerID="a62794f8f51434f8542ab507552bda721d872e0253c6f0174608066b9ad5f2ba" exitCode=0 Nov 02 07:53:45.523176 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.522760 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" event={"ID":"a14a0685-a3b9-4b26-abe8-d5ac282cd653","Type":"ContainerDied","Data":"a62794f8f51434f8542ab507552bda721d872e0253c6f0174608066b9ad5f2ba"} Nov 02 07:53:45.525686 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.525547 5899 generic.go:356] "Generic (PLEG): container finished" podID="5882a478-2c3b-4ead-976c-cc4231e32e12" containerID="976e2190a5f68d001b0914d8207e5386f1536f95bfeb671543b168aef40bddbe" exitCode=0 Nov 02 07:53:45.525686 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.525613 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" event={"ID":"5882a478-2c3b-4ead-976c-cc4231e32e12","Type":"ContainerDied","Data":"976e2190a5f68d001b0914d8207e5386f1536f95bfeb671543b168aef40bddbe"} Nov 02 07:53:45.548994 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.548882 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:45.605790 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.605759 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:53:45.647309 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.647285 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:45.690748 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.690717 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-kube-api-access-59tzt\" (UniqueName: \"kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-kube-api-access-59tzt\") pod \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " Nov 02 07:53:45.690938 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.690783 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-socket\") pod \"5882a478-2c3b-4ead-976c-cc4231e32e12\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " Nov 02 07:53:45.690938 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.690818 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-podinfo\") pod \"5882a478-2c3b-4ead-976c-cc4231e32e12\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " Nov 02 07:53:45.690938 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.690881 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-podinfo\") pod \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " Nov 02 07:53:45.690938 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.690902 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istiod-ca-cert\") pod \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " Nov 02 07:53:45.690938 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.690929 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-token\" (UniqueName: \"kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-token\") pod \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " Nov 02 07:53:45.691243 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.690979 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-data\") pod \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " Nov 02 07:53:45.691243 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.691001 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5882a478-2c3b-4ead-976c-cc4231e32e12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5882a478-2c3b-4ead-976c-cc4231e32e12-istiod-ca-cert\") pod \"5882a478-2c3b-4ead-976c-cc4231e32e12\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " Nov 02 07:53:45.691397 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.691370 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-socket" pod "5882a478-2c3b-4ead-976c-cc4231e32e12" (UID: "5882a478-2c3b-4ead-976c-cc4231e32e12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.691468 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.691433 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istiod-ca-cert" pod "a14a0685-a3b9-4b26-abe8-d5ac282cd653" (UID: "a14a0685-a3b9-4b26-abe8-d5ac282cd653"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:53:45.691622 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.691600 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-data" pod "a14a0685-a3b9-4b26-abe8-d5ac282cd653" (UID: "a14a0685-a3b9-4b26-abe8-d5ac282cd653"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.691961 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.691927 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-certs" pod "5882a478-2c3b-4ead-976c-cc4231e32e12" (UID: "5882a478-2c3b-4ead-976c-cc4231e32e12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.691961 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.691855 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5882a478-2c3b-4ead-976c-cc4231e32e12-istiod-ca-cert" pod "5882a478-2c3b-4ead-976c-cc4231e32e12" (UID: "5882a478-2c3b-4ead-976c-cc4231e32e12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:53:45.692121 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.692055 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-certs\") pod \"5882a478-2c3b-4ead-976c-cc4231e32e12\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " Nov 02 07:53:45.692205 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.692121 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-credential-socket\") pod \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " Nov 02 07:53:45.692399 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.692359 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-credential-socket" pod "a14a0685-a3b9-4b26-abe8-d5ac282cd653" (UID: "a14a0685-a3b9-4b26-abe8-d5ac282cd653"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.692484 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.692431 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-certs" pod "a14a0685-a3b9-4b26-abe8-d5ac282cd653" (UID: "a14a0685-a3b9-4b26-abe8-d5ac282cd653"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.692734 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.692711 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-certs\") pod \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " Nov 02 07:53:45.692859 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.692841 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-token\" (UniqueName: \"kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-token\") pod \"5882a478-2c3b-4ead-976c-cc4231e32e12\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " Nov 02 07:53:45.692939 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.692914 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-data\") pod \"5882a478-2c3b-4ead-976c-cc4231e32e12\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " Nov 02 07:53:45.693072 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.692946 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-socket\") pod \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " Nov 02 07:53:45.693072 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.692980 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-credential-socket\") pod \"5882a478-2c3b-4ead-976c-cc4231e32e12\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " Nov 02 07:53:45.693072 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.693043 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-envoy\") pod \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\" (UID: \"a14a0685-a3b9-4b26-abe8-d5ac282cd653\") " Nov 02 07:53:45.693322 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.693078 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-envoy\") pod \"5882a478-2c3b-4ead-976c-cc4231e32e12\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " Nov 02 07:53:45.693322 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.693133 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-kube-api-access-czh5r\" (UniqueName: \"kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-kube-api-access-czh5r\") pod \"5882a478-2c3b-4ead-976c-cc4231e32e12\" (UID: \"5882a478-2c3b-4ead-976c-cc4231e32e12\") " Nov 02 07:53:45.693515 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.693423 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-credential-socket" pod "5882a478-2c3b-4ead-976c-cc4231e32e12" (UID: "5882a478-2c3b-4ead-976c-cc4231e32e12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.693699 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-data" pod "5882a478-2c3b-4ead-976c-cc4231e32e12" (UID: "5882a478-2c3b-4ead-976c-cc4231e32e12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.693791 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-socket" pod "a14a0685-a3b9-4b26-abe8-d5ac282cd653" (UID: "a14a0685-a3b9-4b26-abe8-d5ac282cd653"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.693852 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-podinfo" pod "5882a478-2c3b-4ead-976c-cc4231e32e12" (UID: "5882a478-2c3b-4ead-976c-cc4231e32e12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.693929 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-token" pod "a14a0685-a3b9-4b26-abe8-d5ac282cd653" (UID: "a14a0685-a3b9-4b26-abe8-d5ac282cd653"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.693966 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.693981 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.693993 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.694006 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.694018 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.694026 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.694034 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.694042 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.694049 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5882a478-2c3b-4ead-976c-cc4231e32e12-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.694057 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.694117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.694065 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.695461 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.695426 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-envoy" pod "5882a478-2c3b-4ead-976c-cc4231e32e12" (UID: "5882a478-2c3b-4ead-976c-cc4231e32e12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.695818 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.695800 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-kube-api-access-czh5r" pod "5882a478-2c3b-4ead-976c-cc4231e32e12" (UID: "5882a478-2c3b-4ead-976c-cc4231e32e12"). InnerVolumeSpecName "kube-api-access-czh5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:53:45.696115 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.696085 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-kube-api-access-59tzt" pod "a14a0685-a3b9-4b26-abe8-d5ac282cd653" (UID: "a14a0685-a3b9-4b26-abe8-d5ac282cd653"). InnerVolumeSpecName "kube-api-access-59tzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:53:45.696115 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.696100 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:45.696293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.696114 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-token" pod "5882a478-2c3b-4ead-976c-cc4231e32e12" (UID: "5882a478-2c3b-4ead-976c-cc4231e32e12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:53:45.696293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.696184 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-podinfo" pod "a14a0685-a3b9-4b26-abe8-d5ac282cd653" (UID: "a14a0685-a3b9-4b26-abe8-d5ac282cd653"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:53:45.697358 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.697339 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-envoy" pod "a14a0685-a3b9-4b26-abe8-d5ac282cd653" (UID: "a14a0685-a3b9-4b26-abe8-d5ac282cd653"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.795133 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.795099 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-certs\") pod \"95439f08-2283-44e9-a424-bb6f9e45fef0\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " Nov 02 07:53:45.795360 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.795221 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-podinfo\") pod \"95439f08-2283-44e9-a424-bb6f9e45fef0\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " Nov 02 07:53:45.795360 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.795253 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95439f08-2283-44e9-a424-bb6f9e45fef0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95439f08-2283-44e9-a424-bb6f9e45fef0-istiod-ca-cert\") pod \"95439f08-2283-44e9-a424-bb6f9e45fef0\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " Nov 02 07:53:45.795360 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.795301 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-credential-socket\") pod \"95439f08-2283-44e9-a424-bb6f9e45fef0\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " Nov 02 07:53:45.795529 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.795357 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-kube-api-access-zvxqp\" (UniqueName: \"kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-kube-api-access-zvxqp\") pod \"95439f08-2283-44e9-a424-bb6f9e45fef0\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " Nov 02 07:53:45.795529 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.795371 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-certs" pod "95439f08-2283-44e9-a424-bb6f9e45fef0" (UID: "95439f08-2283-44e9-a424-bb6f9e45fef0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.795529 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.795448 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-data\") pod \"95439f08-2283-44e9-a424-bb6f9e45fef0\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " Nov 02 07:53:45.795529 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.795489 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-token\" (UniqueName: \"kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-token\") pod \"95439f08-2283-44e9-a424-bb6f9e45fef0\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " Nov 02 07:53:45.796053 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.795629 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-credential-socket" pod "95439f08-2283-44e9-a424-bb6f9e45fef0" (UID: "95439f08-2283-44e9-a424-bb6f9e45fef0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.796053 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.795744 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-data" pod "95439f08-2283-44e9-a424-bb6f9e45fef0" (UID: "95439f08-2283-44e9-a424-bb6f9e45fef0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.796053 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.795968 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95439f08-2283-44e9-a424-bb6f9e45fef0-istiod-ca-cert" pod "95439f08-2283-44e9-a424-bb6f9e45fef0" (UID: "95439f08-2283-44e9-a424-bb6f9e45fef0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:53:45.796053 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796047 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-envoy\") pod \"95439f08-2283-44e9-a424-bb6f9e45fef0\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " Nov 02 07:53:45.796284 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796090 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-socket\") pod \"95439f08-2283-44e9-a424-bb6f9e45fef0\" (UID: \"95439f08-2283-44e9-a424-bb6f9e45fef0\") " Nov 02 07:53:45.796355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796336 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-socket" pod "95439f08-2283-44e9-a424-bb6f9e45fef0" (UID: "95439f08-2283-44e9-a424-bb6f9e45fef0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.796781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796757 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.796874 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796786 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.796874 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796802 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czh5r\" (UniqueName: \"kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-kube-api-access-czh5r\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.796874 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796839 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.796874 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796852 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59tzt\" (UniqueName: \"kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-kube-api-access-59tzt\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.796874 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796865 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.796874 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796877 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a14a0685-a3b9-4b26-abe8-d5ac282cd653-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.797269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796888 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.797269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796901 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.797269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796913 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95439f08-2283-44e9-a424-bb6f9e45fef0-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.797269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796924 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.797269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.796935 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5882a478-2c3b-4ead-976c-cc4231e32e12-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.797840 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.797800 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-kube-api-access-zvxqp" pod "95439f08-2283-44e9-a424-bb6f9e45fef0" (UID: "95439f08-2283-44e9-a424-bb6f9e45fef0"). InnerVolumeSpecName "kube-api-access-zvxqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:53:45.798404 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.798377 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-podinfo" pod "95439f08-2283-44e9-a424-bb6f9e45fef0" (UID: "95439f08-2283-44e9-a424-bb6f9e45fef0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:53:45.798912 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.798886 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-token" pod "95439f08-2283-44e9-a424-bb6f9e45fef0" (UID: "95439f08-2283-44e9-a424-bb6f9e45fef0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:53:45.799071 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.799050 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-envoy" pod "95439f08-2283-44e9-a424-bb6f9e45fef0" (UID: "95439f08-2283-44e9-a424-bb6f9e45fef0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:53:45.898345 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.898301 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.898345 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.898337 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvxqp\" (UniqueName: \"kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-kube-api-access-zvxqp\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.898345 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.898354 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:45.898716 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:45.898366 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95439f08-2283-44e9-a424-bb6f9e45fef0-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:53:46.181966 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.181881 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:46.338715 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.338676 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:46.534704 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.534665 5899 generic.go:356] "Generic (PLEG): container finished" podID="95439f08-2283-44e9-a424-bb6f9e45fef0" containerID="72c1e62303e1e2f06f9100a9745401fe60b8c8083254fea0915e816f29a37636" exitCode=0 Nov 02 07:53:46.535233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.534714 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" Nov 02 07:53:46.535233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.534746 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" event={"ID":"95439f08-2283-44e9-a424-bb6f9e45fef0","Type":"ContainerDied","Data":"72c1e62303e1e2f06f9100a9745401fe60b8c8083254fea0915e816f29a37636"} Nov 02 07:53:46.535233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.534801 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp" event={"ID":"95439f08-2283-44e9-a424-bb6f9e45fef0","Type":"ContainerDied","Data":"ed507e6e94f4f07b6f7248c6344fc8f066b021b9f3a5a817be49aeaa4923ff20"} Nov 02 07:53:46.535233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.534830 5899 scope.go:122] "RemoveContainer" containerID="72c1e62303e1e2f06f9100a9745401fe60b8c8083254fea0915e816f29a37636" Nov 02 07:53:46.536758 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.536739 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" Nov 02 07:53:46.536880 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.536749 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f" event={"ID":"a14a0685-a3b9-4b26-abe8-d5ac282cd653","Type":"ContainerDied","Data":"5defe3d25201ef2b64b91bdcd66f36c0181a858bd5dccb392dd587b456b29b8a"} Nov 02 07:53:46.538410 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.538389 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" event={"ID":"5882a478-2c3b-4ead-976c-cc4231e32e12","Type":"ContainerDied","Data":"1a9e844e0b45f00f269e0e3426016864ea8c832be825a4855aab969570cbbec2"} Nov 02 07:53:46.538555 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.538403 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5" Nov 02 07:53:46.548576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.548553 5899 scope.go:122] "RemoveContainer" containerID="72c1e62303e1e2f06f9100a9745401fe60b8c8083254fea0915e816f29a37636" Nov 02 07:53:46.548961 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:53:46.548930 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c1e62303e1e2f06f9100a9745401fe60b8c8083254fea0915e816f29a37636\": container with ID starting with 72c1e62303e1e2f06f9100a9745401fe60b8c8083254fea0915e816f29a37636 not found: ID does not exist" containerID="72c1e62303e1e2f06f9100a9745401fe60b8c8083254fea0915e816f29a37636" Nov 02 07:53:46.549072 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.548970 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c1e62303e1e2f06f9100a9745401fe60b8c8083254fea0915e816f29a37636"} err="failed to get container status \"72c1e62303e1e2f06f9100a9745401fe60b8c8083254fea0915e816f29a37636\": rpc error: code = NotFound desc = could not find container \"72c1e62303e1e2f06f9100a9745401fe60b8c8083254fea0915e816f29a37636\": container with ID starting with 72c1e62303e1e2f06f9100a9745401fe60b8c8083254fea0915e816f29a37636 not found: ID does not exist" Nov 02 07:53:46.549072 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.549023 5899 scope.go:122] "RemoveContainer" containerID="a62794f8f51434f8542ab507552bda721d872e0253c6f0174608066b9ad5f2ba" Nov 02 07:53:46.559244 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.559226 5899 scope.go:122] "RemoveContainer" containerID="976e2190a5f68d001b0914d8207e5386f1536f95bfeb671543b168aef40bddbe" Nov 02 07:53:46.586418 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.586394 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5"] Nov 02 07:53:46.598017 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.597993 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-x87j5"] Nov 02 07:53:46.623850 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.623816 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f"] Nov 02 07:53:46.628032 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.628008 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-2q86f"] Nov 02 07:53:46.660953 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.660929 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp"] Nov 02 07:53:46.667096 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:46.667075 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-lhpjp"] Nov 02 07:53:47.370890 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:47.370853 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:47.820068 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:47.820035 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5882a478-2c3b-4ead-976c-cc4231e32e12" path="/var/lib/kubelet/pods/5882a478-2c3b-4ead-976c-cc4231e32e12/volumes" Nov 02 07:53:47.823638 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:47.821270 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95439f08-2283-44e9-a424-bb6f9e45fef0" path="/var/lib/kubelet/pods/95439f08-2283-44e9-a424-bb6f9e45fef0/volumes" Nov 02 07:53:47.823638 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:47.822028 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a14a0685-a3b9-4b26-abe8-d5ac282cd653" path="/var/lib/kubelet/pods/a14a0685-a3b9-4b26-abe8-d5ac282cd653/volumes" Nov 02 07:53:48.053921 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:48.053878 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:48.216359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:48.216263 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:48.994575 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:48.994545 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:49.158964 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:49.158927 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:49.328245 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:49.317667 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:51.154883 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:51.154845 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:52.533933 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:52.533895 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:52.987076 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:52.987038 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:53.141005 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:53.140941 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:53.301241 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:53.284604 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:54.744328 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:54.744295 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:54.952874 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:54.952842 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:55.091944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:55.091861 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:55.735757 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:55.735721 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:57.537944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:57.537871 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:57.704663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:57.704558 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:59.307323 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:59.307265 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:59.449563 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:59.449527 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:53:59.614349 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:53:59.614253 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:01.651910 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:01.651871 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:04.018510 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:04.018475 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:04.235137 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:04.235104 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:04.378425 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:04.378336 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:05.702904 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:05.702867 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:05.838567 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:05.838528 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:06.461788 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:06.461751 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:06.599915 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:06.599881 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:06.743459 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:06.743428 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:07.140993 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:07.140910 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:07.554130 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:07.554098 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:08.402347 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:08.402312 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:08.860749 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:08.860716 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:09.003992 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:09.003950 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:09.148603 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:09.148526 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:10.897688 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:10.897654 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:12.712910 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:12.712877 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:13.147767 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:54:13.147730 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:54:14.287391 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:14.287354 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:14.523480 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:14.523447 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:16.297956 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:16.297907 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:17.988661 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:17.988628 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:18.143317 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.143281 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j"] Nov 02 07:54:18.144071 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.144046 5899 cpu_manager.go:414] "containerMap: removing container" podUID="5882a478-2c3b-4ead-976c-cc4231e32e12" containerName="istio-proxy" Nov 02 07:54:18.144261 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.144236 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5882a478-2c3b-4ead-976c-cc4231e32e12" containerName="istio-proxy" Nov 02 07:54:18.144469 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.144300 5899 cpu_manager.go:414] "containerMap: removing container" podUID="a14a0685-a3b9-4b26-abe8-d5ac282cd653" containerName="istio-proxy" Nov 02 07:54:18.144469 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.144309 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14a0685-a3b9-4b26-abe8-d5ac282cd653" containerName="istio-proxy" Nov 02 07:54:18.144469 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.144344 5899 cpu_manager.go:414] "containerMap: removing container" podUID="95439f08-2283-44e9-a424-bb6f9e45fef0" containerName="istio-proxy" Nov 02 07:54:18.144469 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.144351 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95439f08-2283-44e9-a424-bb6f9e45fef0" containerName="istio-proxy" Nov 02 07:54:18.144746 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.144520 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="95439f08-2283-44e9-a424-bb6f9e45fef0" containerName="istio-proxy" Nov 02 07:54:18.144746 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.144542 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="5882a478-2c3b-4ead-976c-cc4231e32e12" containerName="istio-proxy" Nov 02 07:54:18.144746 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.144557 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14a0685-a3b9-4b26-abe8-d5ac282cd653" containerName="istio-proxy" Nov 02 07:54:18.148743 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.148720 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.172198 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.172155 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j"] Nov 02 07:54:18.236820 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.236784 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.236820 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.236823 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.237094 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.236968 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqvx6\" (UniqueName: \"kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-kube-api-access-kqvx6\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.237094 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.237064 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.237245 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.237094 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122a1a15-edb5-43b0-a9fe-9d95381ee889-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.237245 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.237201 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.237356 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.237302 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.237414 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.237365 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.237414 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.237390 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339104 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339008 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339104 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339052 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339104 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339091 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqvx6\" (UniqueName: \"kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-kube-api-access-kqvx6\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339138 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339193 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122a1a15-edb5-43b0-a9fe-9d95381ee889-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339225 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339315 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339372 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339391 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339762 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339568 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339819 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339771 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339819 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339786 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339929 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339833 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.339986 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.339964 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122a1a15-edb5-43b0-a9fe-9d95381ee889-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.341576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.341555 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.341809 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.341791 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.362257 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.362232 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.362403 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.362385 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqvx6\" (UniqueName: \"kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-kube-api-access-kqvx6\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.469325 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.469291 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:18.784961 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.784931 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j"] Nov 02 07:54:18.786650 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:54:18.786624 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122a1a15_edb5_43b0_a9fe_9d95381ee889.slice/crio-ebfd941475a33285a8b239ae9298ca48066adf20c9704883d572236e88794153 WatchSource:0}: Error finding container ebfd941475a33285a8b239ae9298ca48066adf20c9704883d572236e88794153: Status 404 returned error can't find the container with id ebfd941475a33285a8b239ae9298ca48066adf20c9704883d572236e88794153 Nov 02 07:54:18.789213 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.789147 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:54:18.789312 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.789259 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:54:18.789312 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:18.789306 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 07:54:19.685257 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:19.685218 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:19.793014 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:19.792980 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" event={"ID":"122a1a15-edb5-43b0-a9fe-9d95381ee889","Type":"ContainerStarted","Data":"4160f017a3fe59abec8cf7d293b676dd701652149bc95a968dab736070245a71"} Nov 02 07:54:19.793251 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:19.793042 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" event={"ID":"122a1a15-edb5-43b0-a9fe-9d95381ee889","Type":"ContainerStarted","Data":"ebfd941475a33285a8b239ae9298ca48066adf20c9704883d572236e88794153"} Nov 02 07:54:19.881270 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:19.880535 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" podStartSLOduration=1.880512725 podStartE2EDuration="1.880512725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:19.853577679 +0000 UTC m=+1760.151858503" watchObservedRunningTime="2026-11-02 07:54:19.880512725 +0000 UTC m=+1760.178793551" Nov 02 07:54:20.479273 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:20.479231 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:20.480383 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:20.480351 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 07:54:20.480535 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:20.480421 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" podUID="122a1a15-edb5-43b0-a9fe-9d95381ee889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 07:54:21.414667 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.414624 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn"] Nov 02 07:54:21.419287 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.419267 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:21.453236 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.448428 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-e4c131-serving-cert\"" Nov 02 07:54:21.453236 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.451081 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn"] Nov 02 07:54:21.482012 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.481978 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 07:54:21.482298 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.482271 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" podUID="122a1a15-edb5-43b0-a9fe-9d95381ee889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 07:54:21.514964 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.514927 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:21.515194 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.515140 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4e434e-62cc-4239-8e4b-d5a319dd963f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:21.515290 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.515268 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:21.515348 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.515302 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bzcq\" (UniqueName: \"kubernetes.io/projected/df4e434e-62cc-4239-8e4b-d5a319dd963f-kube-api-access-8bzcq\") pod \"maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:21.548499 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.548464 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:21.617899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.616958 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4e434e-62cc-4239-8e4b-d5a319dd963f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:21.617899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.617059 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:21.617899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.617088 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bzcq\" (UniqueName: \"kubernetes.io/projected/df4e434e-62cc-4239-8e4b-d5a319dd963f-kube-api-access-8bzcq\") pod \"maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:21.617899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.617233 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:21.621716 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.621686 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4e434e-62cc-4239-8e4b-d5a319dd963f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:21.621889 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:54:21.621872 5899 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-serving-cert: secret "maas-api-e2e-shared-a-w1-e4c131-serving-cert" not found Nov 02 07:54:21.621955 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:54:21.621945 5899 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-maas-api-tls podName:df4e434e-62cc-4239-8e4b-d5a319dd963f nodeName:}" failed. No retries permitted until 2026-11-02 07:54:22.121925702 +0000 UTC m=+1762.420206520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-maas-api-tls") pod "maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" (UID: "df4e434e-62cc-4239-8e4b-d5a319dd963f") : secret "maas-api-e2e-shared-a-w1-e4c131-serving-cert" not found Nov 02 07:54:21.622390 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.622359 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:21.645884 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.645855 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bzcq\" (UniqueName: \"kubernetes.io/projected/df4e434e-62cc-4239-8e4b-d5a319dd963f-kube-api-access-8bzcq\") pod \"maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:21.685635 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.685463 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:21.850793 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:21.850762 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:22.123232 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:22.123197 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:22.125577 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:22.125556 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:22.333733 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:22.333702 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:22.501082 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:22.501047 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 07:54:22.510329 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:22.510281 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" podUID="122a1a15-edb5-43b0-a9fe-9d95381ee889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 07:54:22.697317 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:22.697090 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn"] Nov 02 07:54:22.700014 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:54:22.699111 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4e434e_62cc_4239_8e4b_d5a319dd963f.slice/crio-d68dde38127d2566cfffb3242ab35b9419c57699a456c2512531f30813597290 WatchSource:0}: Error finding container d68dde38127d2566cfffb3242ab35b9419c57699a456c2512531f30813597290: Status 404 returned error can't find the container with id d68dde38127d2566cfffb3242ab35b9419c57699a456c2512531f30813597290 Nov 02 07:54:22.701186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:22.701143 5899 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 07:54:22.826112 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:22.826030 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" event={"ID":"df4e434e-62cc-4239-8e4b-d5a319dd963f","Type":"ContainerStarted","Data":"d68dde38127d2566cfffb3242ab35b9419c57699a456c2512531f30813597290"} Nov 02 07:54:23.480303 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:23.480262 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 07:54:23.480467 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:23.480326 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" podUID="122a1a15-edb5-43b0-a9fe-9d95381ee889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 07:54:23.790180 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:23.790088 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:23.838130 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:23.838092 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" event={"ID":"df4e434e-62cc-4239-8e4b-d5a319dd963f","Type":"ContainerStarted","Data":"7d74de8792a6173a2ee394bbf370a8e76fb92b39f5a5de007aa2248d07fd5eda"} Nov 02 07:54:23.838330 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:23.838152 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:23.886898 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:23.884599 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" podStartSLOduration=2.3878551359999998 podStartE2EDuration="2.884581033s" totalImagesPullingTime="496.725897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:23.881627724 +0000 UTC m=+1764.179908552" watchObservedRunningTime="2026-11-02 07:54:23.884581033 +0000 UTC m=+1764.182861859" Nov 02 07:54:24.477186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:24.473893 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 07:54:24.477186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:24.473969 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" podUID="122a1a15-edb5-43b0-a9fe-9d95381ee889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 07:54:25.476198 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:25.470403 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 07:54:25.476198 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:25.470465 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" podUID="122a1a15-edb5-43b0-a9fe-9d95381ee889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 07:54:25.923308 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:25.923274 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:26.712557 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:26.712527 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:26.713087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:26.712975 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:26.713820 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:26.713796 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:54:27.824127 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:27.824073 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:28.007802 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:28.007773 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:28.141789 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:28.141677 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:28.809661 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:28.809628 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:29.778128 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:29.778098 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:29.851396 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:29.851362 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 07:54:29.931388 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:29.931339 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:30.677233 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:30.677196 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:31.070355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:31.070322 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:31.238258 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:31.238214 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:31.625597 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:31.625561 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:32.762997 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:32.762963 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:33.281815 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:33.281788 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:33.508987 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:33.508945 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:33.653821 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:33.653537 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:34.893990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:34.893894 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:35.821678 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:35.821648 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:35.971661 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:35.971629 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:36.277725 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:36.277687 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:36.725755 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:36.725661 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:36.864255 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:36.864213 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:37.013514 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:37.013463 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:37.174501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:37.174466 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:37.364673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:37.364640 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:38.405342 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:38.405303 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:39.644665 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:39.644614 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:39.790125 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:39.790096 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:40.766334 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:40.766306 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:41.043038 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:41.042948 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:41.298536 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:41.298445 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:41.479351 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:41.479314 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:41.624095 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:41.623990 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:43.031208 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:43.031121 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:43.141888 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:54:43.141851 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:54:43.183064 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:43.183035 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:43.330549 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:43.330408 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:44.519521 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:44.519479 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr"] Nov 02 07:54:44.520010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:44.519733 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" containerID="cri-o://4c2197862016cfc5808219da504f7670accd4ed1e3814b84f3ee0b2c63f4f3a0" gracePeriod=30 Nov 02 07:54:45.383552 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:45.383519 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:47.213427 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:47.213390 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:49.081880 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.081841 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:49.788343 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.788284 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:54:49.849965 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.849925 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-kube-api-access-m7dr8\" (UniqueName: \"kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-kube-api-access-m7dr8\") pod \"d0257772-1251-45b3-8c95-ffbe53bda5f3\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " Nov 02 07:54:49.850146 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.850015 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-credential-socket\") pod \"d0257772-1251-45b3-8c95-ffbe53bda5f3\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " Nov 02 07:54:49.850146 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.850093 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-data\") pod \"d0257772-1251-45b3-8c95-ffbe53bda5f3\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " Nov 02 07:54:49.850146 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.850136 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-socket\") pod \"d0257772-1251-45b3-8c95-ffbe53bda5f3\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " Nov 02 07:54:49.850359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.850194 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-podinfo\") pod \"d0257772-1251-45b3-8c95-ffbe53bda5f3\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " Nov 02 07:54:49.850359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.850265 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0257772-1251-45b3-8c95-ffbe53bda5f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0257772-1251-45b3-8c95-ffbe53bda5f3-istiod-ca-cert\") pod \"d0257772-1251-45b3-8c95-ffbe53bda5f3\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " Nov 02 07:54:49.850359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.850292 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-envoy\") pod \"d0257772-1251-45b3-8c95-ffbe53bda5f3\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " Nov 02 07:54:49.850644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.850389 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-credential-socket" pod "d0257772-1251-45b3-8c95-ffbe53bda5f3" (UID: "d0257772-1251-45b3-8c95-ffbe53bda5f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:54:49.850644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.850438 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-certs\") pod \"d0257772-1251-45b3-8c95-ffbe53bda5f3\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " Nov 02 07:54:49.850644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.850493 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-token\" (UniqueName: \"kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-token\") pod \"d0257772-1251-45b3-8c95-ffbe53bda5f3\" (UID: \"d0257772-1251-45b3-8c95-ffbe53bda5f3\") " Nov 02 07:54:49.850644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.850500 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-socket" pod "d0257772-1251-45b3-8c95-ffbe53bda5f3" (UID: "d0257772-1251-45b3-8c95-ffbe53bda5f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:54:49.851316 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.851287 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-certs" pod "d0257772-1251-45b3-8c95-ffbe53bda5f3" (UID: "d0257772-1251-45b3-8c95-ffbe53bda5f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:54:49.851433 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.851331 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0257772-1251-45b3-8c95-ffbe53bda5f3-istiod-ca-cert" pod "d0257772-1251-45b3-8c95-ffbe53bda5f3" (UID: "d0257772-1251-45b3-8c95-ffbe53bda5f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:54:49.852616 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.852592 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:49.852731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.852621 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:49.852731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.852637 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:49.852731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.852651 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0257772-1251-45b3-8c95-ffbe53bda5f3-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:49.853069 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.853045 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-data" pod "d0257772-1251-45b3-8c95-ffbe53bda5f3" (UID: "d0257772-1251-45b3-8c95-ffbe53bda5f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:54:49.853771 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.853747 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-envoy" pod "d0257772-1251-45b3-8c95-ffbe53bda5f3" (UID: "d0257772-1251-45b3-8c95-ffbe53bda5f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:54:49.853874 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.853826 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-podinfo" pod "d0257772-1251-45b3-8c95-ffbe53bda5f3" (UID: "d0257772-1251-45b3-8c95-ffbe53bda5f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:54:49.854000 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.853970 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-token" pod "d0257772-1251-45b3-8c95-ffbe53bda5f3" (UID: "d0257772-1251-45b3-8c95-ffbe53bda5f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:54:49.854237 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.854204 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-kube-api-access-m7dr8" pod "d0257772-1251-45b3-8c95-ffbe53bda5f3" (UID: "d0257772-1251-45b3-8c95-ffbe53bda5f3"). InnerVolumeSpecName "kube-api-access-m7dr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:54:49.953439 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.953406 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:49.953439 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.953433 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:49.953439 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.953442 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:49.953676 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.953451 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7dr8\" (UniqueName: \"kubernetes.io/projected/d0257772-1251-45b3-8c95-ffbe53bda5f3-kube-api-access-m7dr8\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:49.953676 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:49.953461 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0257772-1251-45b3-8c95-ffbe53bda5f3-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:50.066979 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:50.066897 5899 generic.go:356] "Generic (PLEG): container finished" podID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerID="4c2197862016cfc5808219da504f7670accd4ed1e3814b84f3ee0b2c63f4f3a0" exitCode=0 Nov 02 07:54:50.066979 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:50.066963 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" Nov 02 07:54:50.067199 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:50.066981 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" event={"ID":"d0257772-1251-45b3-8c95-ffbe53bda5f3","Type":"ContainerDied","Data":"4c2197862016cfc5808219da504f7670accd4ed1e3814b84f3ee0b2c63f4f3a0"} Nov 02 07:54:50.067199 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:50.067071 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr" event={"ID":"d0257772-1251-45b3-8c95-ffbe53bda5f3","Type":"ContainerDied","Data":"a8a785e25d8b9efe6de28e03250efaf0b94e0ef75494459449b9da792eec4bd5"} Nov 02 07:54:50.067199 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:50.067094 5899 scope.go:122] "RemoveContainer" containerID="4c2197862016cfc5808219da504f7670accd4ed1e3814b84f3ee0b2c63f4f3a0" Nov 02 07:54:50.080515 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:50.080487 5899 scope.go:122] "RemoveContainer" containerID="4c2197862016cfc5808219da504f7670accd4ed1e3814b84f3ee0b2c63f4f3a0" Nov 02 07:54:50.080894 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:54:50.080871 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2197862016cfc5808219da504f7670accd4ed1e3814b84f3ee0b2c63f4f3a0\": container with ID starting with 4c2197862016cfc5808219da504f7670accd4ed1e3814b84f3ee0b2c63f4f3a0 not found: ID does not exist" containerID="4c2197862016cfc5808219da504f7670accd4ed1e3814b84f3ee0b2c63f4f3a0" Nov 02 07:54:50.081001 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:50.080906 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2197862016cfc5808219da504f7670accd4ed1e3814b84f3ee0b2c63f4f3a0"} err="failed to get container status \"4c2197862016cfc5808219da504f7670accd4ed1e3814b84f3ee0b2c63f4f3a0\": rpc error: code = NotFound desc = could not find container \"4c2197862016cfc5808219da504f7670accd4ed1e3814b84f3ee0b2c63f4f3a0\": container with ID starting with 4c2197862016cfc5808219da504f7670accd4ed1e3814b84f3ee0b2c63f4f3a0 not found: ID does not exist" Nov 02 07:54:50.123211 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:50.123172 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr"] Nov 02 07:54:50.129423 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:50.129393 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-xtlwr"] Nov 02 07:54:51.204386 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:51.204348 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:51.763429 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:51.763394 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx"] Nov 02 07:54:51.763669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:51.763643 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerName="istio-proxy" containerID="cri-o://89d12d477780c3826cd25cb1dc1bc614837c2ed15652cb30bfc61625932fa251" gracePeriod=30 Nov 02 07:54:51.822503 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:51.822468 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" path="/var/lib/kubelet/pods/d0257772-1251-45b3-8c95-ffbe53bda5f3/volumes" Nov 02 07:54:53.147384 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:53.147353 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:53.451882 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:53.451806 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:53.912606 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:53.912568 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:54.390524 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:54.390490 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:55.007039 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:55.007000 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:55.219330 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:55.219292 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:56.938019 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:56.937951 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:57.079895 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.079862 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:54:57.116622 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.116591 5899 generic.go:356] "Generic (PLEG): container finished" podID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerID="89d12d477780c3826cd25cb1dc1bc614837c2ed15652cb30bfc61625932fa251" exitCode=0 Nov 02 07:54:57.116798 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.116638 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" Nov 02 07:54:57.116798 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.116670 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" event={"ID":"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769","Type":"ContainerDied","Data":"89d12d477780c3826cd25cb1dc1bc614837c2ed15652cb30bfc61625932fa251"} Nov 02 07:54:57.116798 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.116760 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx" event={"ID":"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769","Type":"ContainerDied","Data":"12c3fcec1f90a61c80652ff09a014adfcd43a257118e1839c395f03b8189894f"} Nov 02 07:54:57.116798 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.116784 5899 scope.go:122] "RemoveContainer" containerID="89d12d477780c3826cd25cb1dc1bc614837c2ed15652cb30bfc61625932fa251" Nov 02 07:54:57.130994 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.130973 5899 scope.go:122] "RemoveContainer" containerID="89d12d477780c3826cd25cb1dc1bc614837c2ed15652cb30bfc61625932fa251" Nov 02 07:54:57.131293 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:54:57.131268 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d12d477780c3826cd25cb1dc1bc614837c2ed15652cb30bfc61625932fa251\": container with ID starting with 89d12d477780c3826cd25cb1dc1bc614837c2ed15652cb30bfc61625932fa251 not found: ID does not exist" containerID="89d12d477780c3826cd25cb1dc1bc614837c2ed15652cb30bfc61625932fa251" Nov 02 07:54:57.131371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.131307 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d12d477780c3826cd25cb1dc1bc614837c2ed15652cb30bfc61625932fa251"} err="failed to get container status \"89d12d477780c3826cd25cb1dc1bc614837c2ed15652cb30bfc61625932fa251\": rpc error: code = NotFound desc = could not find container \"89d12d477780c3826cd25cb1dc1bc614837c2ed15652cb30bfc61625932fa251\": container with ID starting with 89d12d477780c3826cd25cb1dc1bc614837c2ed15652cb30bfc61625932fa251 not found: ID does not exist" Nov 02 07:54:57.165153 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.165118 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-certs\") pod \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " Nov 02 07:54:57.165334 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.165224 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-envoy\") pod \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " Nov 02 07:54:57.165334 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.165272 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-kube-api-access-s4hfl\" (UniqueName: \"kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-kube-api-access-s4hfl\") pod \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " Nov 02 07:54:57.165334 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.165328 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-socket\") pod \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " Nov 02 07:54:57.165492 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.165367 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-data\") pod \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " Nov 02 07:54:57.165606 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.165478 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-certs" pod "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" (UID: "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:54:57.165606 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.165528 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-credential-socket\") pod \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " Nov 02 07:54:57.165606 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.165575 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-token\" (UniqueName: \"kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-token\") pod \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " Nov 02 07:54:57.165752 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.165662 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istiod-ca-cert\") pod \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " Nov 02 07:54:57.165752 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.165715 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-podinfo\") pod \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\" (UID: \"67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769\") " Nov 02 07:54:57.165855 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.165764 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-credential-socket" pod "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" (UID: "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:54:57.166132 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.166101 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istiod-ca-cert" pod "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" (UID: "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:54:57.166440 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.166417 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:57.166511 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.166444 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:57.166511 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.166459 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:57.168271 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.168244 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-socket" pod "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" (UID: "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:54:57.170420 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.170379 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-token" pod "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" (UID: "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:54:57.170585 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.170484 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-podinfo" pod "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" (UID: "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:54:57.170585 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.170484 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-kube-api-access-s4hfl" pod "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" (UID: "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769"). InnerVolumeSpecName "kube-api-access-s4hfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:54:57.172598 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.172572 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-envoy" pod "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" (UID: "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:54:57.175393 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.175371 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-data" pod "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" (UID: "67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:54:57.268059 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.268028 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:57.268059 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.268056 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:57.268059 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.268066 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:57.268335 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.268075 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4hfl\" (UniqueName: \"kubernetes.io/projected/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-kube-api-access-s4hfl\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:57.268335 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.268086 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:57.268335 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.268094 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:54:57.462191 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.462146 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx"] Nov 02 07:54:57.473803 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.473771 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-n6cpx"] Nov 02 07:54:57.820115 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:57.820079 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" path="/var/lib/kubelet/pods/67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769/volumes" Nov 02 07:54:58.467018 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:58.466966 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:58.617850 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:58.617820 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:58.806875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:58.806838 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:54:58.953181 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:54:58.952473 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:00.839663 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.839630 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs"] Nov 02 07:55:00.840222 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.840206 5899 cpu_manager.go:414] "containerMap: removing container" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" Nov 02 07:55:00.840279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.840222 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" Nov 02 07:55:00.840279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.840246 5899 cpu_manager.go:414] "containerMap: removing container" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerName="istio-proxy" Nov 02 07:55:00.840279 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.840251 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerName="istio-proxy" Nov 02 07:55:00.840372 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.840342 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="67cc3ce4-bd3c-426e-9f7b-0ca5ca7cc769" containerName="istio-proxy" Nov 02 07:55:00.840372 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.840358 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0257772-1251-45b3-8c95-ffbe53bda5f3" containerName="istio-proxy" Nov 02 07:55:00.845510 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.845491 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:00.866405 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.866379 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs"] Nov 02 07:55:00.875456 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.875430 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c7a27f-serving-cert\"" Nov 02 07:55:00.891168 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.891140 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:00.915077 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.915049 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55cl9\" (UniqueName: \"kubernetes.io/projected/2b2c6fb1-14a7-4133-9a89-297807894f69-kube-api-access-55cl9\") pod \"maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:00.915920 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.915385 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-metrics-certs\") pod \"maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:00.915920 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.915476 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2c6fb1-14a7-4133-9a89-297807894f69-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:00.915920 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:00.915756 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:01.016879 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:01.016840 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-metrics-certs\") pod \"maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:01.017103 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:01.016896 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2c6fb1-14a7-4133-9a89-297807894f69-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:01.017103 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:01.016962 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:01.017103 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:01.017013 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55cl9\" (UniqueName: \"kubernetes.io/projected/2b2c6fb1-14a7-4133-9a89-297807894f69-kube-api-access-55cl9\") pod \"maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:01.017732 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:01.017706 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2c6fb1-14a7-4133-9a89-297807894f69-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:01.019658 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:01.019629 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:01.019750 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:01.019641 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-metrics-certs\") pod \"maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:01.025301 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:01.025272 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:01.040240 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:01.040203 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55cl9\" (UniqueName: \"kubernetes.io/projected/2b2c6fb1-14a7-4133-9a89-297807894f69-kube-api-access-55cl9\") pod \"maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:01.160081 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:01.160002 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:01.470832 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:01.470742 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs"] Nov 02 07:55:01.479659 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:55:01.479622 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2c6fb1_14a7_4133_9a89_297807894f69.slice/crio-ff59eda2ce2d0165b5cf767b0294105b8163993bdb646e20bb9203ef349fd000 WatchSource:0}: Error finding container ff59eda2ce2d0165b5cf767b0294105b8163993bdb646e20bb9203ef349fd000: Status 404 returned error can't find the container with id ff59eda2ce2d0165b5cf767b0294105b8163993bdb646e20bb9203ef349fd000 Nov 02 07:55:02.004205 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:02.004142 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:02.155100 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:02.155023 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" event={"ID":"2b2c6fb1-14a7-4133-9a89-297807894f69","Type":"ContainerStarted","Data":"15fdb836f3b39a9379162c415efcb28f168fe628caf6b59fa23d31c45b66bb04"} Nov 02 07:55:02.155100 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:02.155081 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" event={"ID":"2b2c6fb1-14a7-4133-9a89-297807894f69","Type":"ContainerStarted","Data":"ff59eda2ce2d0165b5cf767b0294105b8163993bdb646e20bb9203ef349fd000"} Nov 02 07:55:02.155312 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:02.155240 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:02.159242 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:02.159216 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:02.201363 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:02.201301 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" podStartSLOduration=1.818229221 podStartE2EDuration="2.201282401s" totalImagesPullingTime="383.05318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:55:02.197420828 +0000 UTC m=+1802.495701665" watchObservedRunningTime="2026-11-02 07:55:02.201282401 +0000 UTC m=+1802.499563227" Nov 02 07:55:03.121034 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:03.120982 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:04.143300 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:04.143264 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:04.360083 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:04.360045 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:04.973415 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:04.973381 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:06.557984 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.557948 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6"] Nov 02 07:55:06.562402 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.562382 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:06.577390 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.577367 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-3575b9-serving-cert\"" Nov 02 07:55:06.589708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.589675 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:06.589833 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.589726 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:06.589935 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.589914 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:06.590176 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.590140 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxsb7\" (UniqueName: \"kubernetes.io/projected/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-kube-api-access-nxsb7\") pod \"maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:06.592953 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.592929 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6"] Nov 02 07:55:06.690830 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.690792 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:06.691205 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.691155 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:06.691278 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.691225 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:06.691490 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.691449 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:06.691612 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.691560 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxsb7\" (UniqueName: \"kubernetes.io/projected/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-kube-api-access-nxsb7\") pod \"maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:06.692231 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.692206 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:06.693768 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.693741 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:06.693857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.693795 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:06.716604 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.716580 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxsb7\" (UniqueName: \"kubernetes.io/projected/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-kube-api-access-nxsb7\") pod \"maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:06.830004 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.829896 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:06.876796 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:06.876758 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:07.189592 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:07.189523 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6"] Nov 02 07:55:07.195106 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:55:07.195077 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a39cd06_3ed4_4b12_9a75_f0cd30e4160e.slice/crio-c256ecfc04e0a27f1af7fb6f3593646d456aa8a350172b28de6f41fd445ddb8d WatchSource:0}: Error finding container c256ecfc04e0a27f1af7fb6f3593646d456aa8a350172b28de6f41fd445ddb8d: Status 404 returned error can't find the container with id c256ecfc04e0a27f1af7fb6f3593646d456aa8a350172b28de6f41fd445ddb8d Nov 02 07:55:08.170578 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:08.170544 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:55:08.200667 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:08.200634 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" event={"ID":"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e","Type":"ContainerStarted","Data":"05ac40eca641767a93ea1d4cee6e50278de01385697f81f4005c85bf83ab39d6"} Nov 02 07:55:08.200832 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:08.200692 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:08.200832 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:08.200710 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" event={"ID":"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e","Type":"ContainerStarted","Data":"c256ecfc04e0a27f1af7fb6f3593646d456aa8a350172b28de6f41fd445ddb8d"} Nov 02 07:55:08.255595 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:08.255534 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" podStartSLOduration=1.925514786 podStartE2EDuration="2.255511749s" totalImagesPullingTime="329.996963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:55:08.254736358 +0000 UTC m=+1808.553017188" watchObservedRunningTime="2026-11-02 07:55:08.255511749 +0000 UTC m=+1808.553792575" Nov 02 07:55:08.473102 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:08.473018 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:09.061609 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:09.061577 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:10.503864 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:10.503832 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:12.269188 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:12.269138 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:13.141770 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:55:13.141728 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:55:14.009200 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:14.009142 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:14.234340 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:14.234312 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 07:55:15.708437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:15.708392 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:17.441263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:17.441224 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:17.579078 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:17.579046 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:19.203410 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:19.203376 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:20.934239 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:20.934191 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:22.682801 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:22.682766 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:24.225853 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.225807 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46"] Nov 02 07:55:24.230635 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.230602 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.266206 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.257136 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-6929a4-serving-cert\"" Nov 02 07:55:24.266206 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.263871 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46"] Nov 02 07:55:24.350229 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.350190 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:24.359562 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.359534 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.359774 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.359751 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.359835 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.359811 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da87585-b423-4543-b5bd-7c3dc5e55063-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.359944 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.359925 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsw5n\" (UniqueName: \"kubernetes.io/projected/1da87585-b423-4543-b5bd-7c3dc5e55063-kube-api-access-vsw5n\") pod \"maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.461535 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.461492 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.462197 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.462153 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da87585-b423-4543-b5bd-7c3dc5e55063-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.462413 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.462394 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw5n\" (UniqueName: \"kubernetes.io/projected/1da87585-b423-4543-b5bd-7c3dc5e55063-kube-api-access-vsw5n\") pod \"maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.462593 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.462574 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.463468 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.463442 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da87585-b423-4543-b5bd-7c3dc5e55063-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.466401 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.466355 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.479256 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.479192 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.486644 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.486614 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsw5n\" (UniqueName: \"kubernetes.io/projected/1da87585-b423-4543-b5bd-7c3dc5e55063-kube-api-access-vsw5n\") pod \"maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.488297 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.488269 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:24.545129 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.545094 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:24.850118 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:24.850082 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46"] Nov 02 07:55:24.851298 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:55:24.851275 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da87585_b423_4543_b5bd_7c3dc5e55063.slice/crio-7c1fdae4de814606d813193dee9f1207942f25cd7b0ea2c013f141ccdfdada36 WatchSource:0}: Error finding container 7c1fdae4de814606d813193dee9f1207942f25cd7b0ea2c013f141ccdfdada36: Status 404 returned error can't find the container with id 7c1fdae4de814606d813193dee9f1207942f25cd7b0ea2c013f141ccdfdada36 Nov 02 07:55:25.328668 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:25.328632 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" event={"ID":"1da87585-b423-4543-b5bd-7c3dc5e55063","Type":"ContainerStarted","Data":"980dddf1221353074379b66f914a3fefb97439721e48498e1ab80c2e7fadbe1d"} Nov 02 07:55:25.329069 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:25.328703 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" event={"ID":"1da87585-b423-4543-b5bd-7c3dc5e55063","Type":"ContainerStarted","Data":"7c1fdae4de814606d813193dee9f1207942f25cd7b0ea2c013f141ccdfdada36"} Nov 02 07:55:25.329069 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:25.328740 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:25.373086 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:25.373035 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" podStartSLOduration=0.965348339 podStartE2EDuration="1.373020685s" totalImagesPullingTime="407.672346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:55:25.370034893 +0000 UTC m=+1825.668315719" watchObservedRunningTime="2026-11-02 07:55:25.373020685 +0000 UTC m=+1825.671301513" Nov 02 07:55:26.222389 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:26.222357 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:28.003570 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:28.003537 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:29.886702 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:29.886665 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:31.342236 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:31.342207 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 07:55:31.350314 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:31.350282 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:31.903706 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:31.903665 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:32.963447 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:32.963416 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:33.905322 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:33.905290 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:34.182238 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:34.182139 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:34.667544 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:34.667510 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:34.858342 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:34.857152 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:35.839314 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:35.839267 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:37.091091 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:37.091051 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:37.701953 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:37.701914 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:39.130215 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:39.130182 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:39.269336 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:39.269306 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:39.738573 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:39.738530 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:40.833435 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:40.833391 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:40.993676 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:40.993635 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:41.905338 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:41.905294 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:43.141017 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:55:43.140983 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:55:43.621355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:43.621322 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:45.293476 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:45.293430 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:47.015440 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:47.015406 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:48.811463 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:48.811428 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:50.625153 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:50.625100 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:52.409693 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:52.409654 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:54.439001 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:54.438954 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:56.393061 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.393026 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:56.635414 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.635381 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng"] Nov 02 07:55:56.641056 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.641027 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.668677 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.668643 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-32ea65\"/\"openshift-service-ca.crt\"" Nov 02 07:55:56.668960 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.668740 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-32ea65\"/\"kube-root-ca.crt\"" Nov 02 07:55:56.668960 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.668958 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-32ea65\"/\"auth-test-model-32ea65-kserve-self-signed-certs\"" Nov 02 07:55:56.669213 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.668796 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-32ea65\"/\"default-dockercfg-ttphc\"" Nov 02 07:55:56.671864 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.671843 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng"] Nov 02 07:55:56.719899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.719862 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:56.760238 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.760205 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-home\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.760238 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.760244 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-tmp-dir\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.760465 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.760279 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db63785-0b2b-4516-b06d-3df792f2019f-tls-certs\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.760465 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.760412 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-model-cache\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.760534 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.760475 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljl5t\" (UniqueName: \"kubernetes.io/projected/7db63785-0b2b-4516-b06d-3df792f2019f-kube-api-access-ljl5t\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.760568 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.760532 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-dshm\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.861978 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.861927 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-model-cache\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.862194 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.862015 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljl5t\" (UniqueName: \"kubernetes.io/projected/7db63785-0b2b-4516-b06d-3df792f2019f-kube-api-access-ljl5t\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.862194 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.862117 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-dshm\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.862323 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.862287 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-home\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.862398 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.862320 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-tmp-dir\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.862398 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.862359 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db63785-0b2b-4516-b06d-3df792f2019f-tls-certs\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.862398 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.862377 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-model-cache\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.862779 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.862755 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-home\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.862889 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.862841 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-tmp-dir\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.864900 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.864876 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-dshm\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.865015 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.864891 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db63785-0b2b-4516-b06d-3df792f2019f-tls-certs\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.896188 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.896099 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljl5t\" (UniqueName: \"kubernetes.io/projected/7db63785-0b2b-4516-b06d-3df792f2019f-kube-api-access-ljl5t\") pod \"auth-test-model-32ea65-kserve-944bd89c-gpbng\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:56.959690 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:56.958433 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:57.292581 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:57.292547 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng"] Nov 02 07:55:57.294553 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:55:57.294521 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7db63785_0b2b_4516_b06d_3df792f2019f.slice/crio-a890835d3904df1b72ff52717be51cf871100a23bc0cd5a642d67f8b544e3f56 WatchSource:0}: Error finding container a890835d3904df1b72ff52717be51cf871100a23bc0cd5a642d67f8b544e3f56: Status 404 returned error can't find the container with id a890835d3904df1b72ff52717be51cf871100a23bc0cd5a642d67f8b544e3f56 Nov 02 07:55:57.559272 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:57.559174 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" event={"ID":"7db63785-0b2b-4516-b06d-3df792f2019f","Type":"ContainerStarted","Data":"37e107ab874b2155c459cba959f1e2c29739c3065b0fc7f0189b6262a3e0430f"} Nov 02 07:55:57.559272 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:57.559221 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" event={"ID":"7db63785-0b2b-4516-b06d-3df792f2019f","Type":"ContainerStarted","Data":"a890835d3904df1b72ff52717be51cf871100a23bc0cd5a642d67f8b544e3f56"} Nov 02 07:55:57.603960 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:57.603905 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" podStartSLOduration=1.603890228 podStartE2EDuration="1.603890228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:55:57.600905551 +0000 UTC m=+1857.899186375" watchObservedRunningTime="2026-11-02 07:55:57.603890228 +0000 UTC m=+1857.902171053" Nov 02 07:55:57.958999 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:57.958927 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:55:58.289596 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:58.289554 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:55:59.547897 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:55:59.547856 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:00.060550 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:00.060514 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:01.758206 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:01.758127 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:03.547760 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:03.547724 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:04.315576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:04.315541 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:05.305097 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:05.305061 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:06.950802 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:06.950767 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2"] Nov 02 07:56:06.956444 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:06.956420 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:06.959593 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:06.959567 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:56:06.977718 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:06.977652 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:56:06.983496 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:06.983459 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-680c01-serving-cert\"" Nov 02 07:56:06.988324 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:06.988291 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2"] Nov 02 07:56:07.072679 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.072640 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:07.109424 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.109378 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zghcb\" (UniqueName: \"kubernetes.io/projected/d44c845c-5941-41be-baa3-1baa0a2b559e-kube-api-access-zghcb\") pod \"maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:07.109618 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.109514 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-metrics-certs\") pod \"maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:07.109618 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.109546 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44c845c-5941-41be-baa3-1baa0a2b559e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:07.109777 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.109754 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:07.211778 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.211689 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zghcb\" (UniqueName: \"kubernetes.io/projected/d44c845c-5941-41be-baa3-1baa0a2b559e-kube-api-access-zghcb\") pod \"maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:07.211778 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.211767 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-metrics-certs\") pod \"maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:07.212038 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.211789 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44c845c-5941-41be-baa3-1baa0a2b559e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:07.212038 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.211895 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:07.212578 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.212559 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44c845c-5941-41be-baa3-1baa0a2b559e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:07.214372 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.214349 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-metrics-certs\") pod \"maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:07.214565 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.214527 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:07.216466 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.216418 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:07.245220 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.245037 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zghcb\" (UniqueName: \"kubernetes.io/projected/d44c845c-5941-41be-baa3-1baa0a2b559e-kube-api-access-zghcb\") pod \"maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:07.271599 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.271566 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:07.363494 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.363317 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:07.600193 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.600151 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2"] Nov 02 07:56:07.601892 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:56:07.601866 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44c845c_5941_41be_baa3_1baa0a2b559e.slice/crio-8514c7b11a4c68fc91cd835878319ef83a0b9141eb5111b05008efcdd00ad246 WatchSource:0}: Error finding container 8514c7b11a4c68fc91cd835878319ef83a0b9141eb5111b05008efcdd00ad246: Status 404 returned error can't find the container with id 8514c7b11a4c68fc91cd835878319ef83a0b9141eb5111b05008efcdd00ad246 Nov 02 07:56:07.638369 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.638332 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" event={"ID":"d44c845c-5941-41be-baa3-1baa0a2b559e","Type":"ContainerStarted","Data":"8514c7b11a4c68fc91cd835878319ef83a0b9141eb5111b05008efcdd00ad246"} Nov 02 07:56:07.649590 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:07.649561 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 07:56:08.613271 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:08.613236 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:08.648578 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:08.648538 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" event={"ID":"d44c845c-5941-41be-baa3-1baa0a2b559e","Type":"ContainerStarted","Data":"69d254510a3213ef1b437f30a66bab32cc9d6d98f3f90da0860f0fe03ecd98e6"} Nov 02 07:56:08.691683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:08.691623 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" podStartSLOduration=2.346410684 podStartE2EDuration="2.691603769s" totalImagesPullingTime="345.193085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:56:08.690217996 +0000 UTC m=+1868.988498821" watchObservedRunningTime="2026-11-02 07:56:08.691603769 +0000 UTC m=+1868.989884595" Nov 02 07:56:09.071799 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:09.071765 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:09.654995 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:09.654965 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:10.824820 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:10.824778 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:12.625425 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:12.625394 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:13.141227 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:56:13.141190 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:56:14.320632 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:14.320591 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:15.666748 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:15.666719 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:56:16.097264 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:16.097202 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:18.012598 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:18.012542 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:19.751710 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:19.751676 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:21.484443 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:21.484404 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:23.191298 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:23.191265 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:24.892428 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:24.892391 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:26.617005 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:26.616971 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:28.317811 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:28.317777 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:30.394968 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:30.394929 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:31.786739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:31.786701 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:32.134611 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:32.134510 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:32.269992 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:32.269944 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:32.413761 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:32.413663 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:34.307411 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:34.307372 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:36.016206 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:36.016144 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:37.767139 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:37.767103 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:39.512958 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:39.512918 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:40.277245 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:40.277214 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:41.327491 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:41.327459 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:42.509245 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:42.509202 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:43.141302 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:56:43.141263 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:56:43.222111 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:43.222053 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:45.001546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:45.001505 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:46.741528 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:46.741490 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:48.201749 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:48.201710 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:48.340645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:48.340612 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:48.668407 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:48.668367 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:49.355123 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:49.355088 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:50.648484 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:50.648443 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:52.494317 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:52.494282 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:53.851833 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:53.851794 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs"] Nov 02 07:56:53.852317 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:53.852070 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" podUID="2b2c6fb1-14a7-4133-9a89-297807894f69" containerName="maas-api" containerID="cri-o://15fdb836f3b39a9379162c415efcb28f168fe628caf6b59fa23d31c45b66bb04" gracePeriod=30 Nov 02 07:56:53.947949 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:53.947912 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:54.251424 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.251386 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:56:54.372809 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.372767 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-maas-api-tls\") pod \"2b2c6fb1-14a7-4133-9a89-297807894f69\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " Nov 02 07:56:54.373457 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.373033 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2c6fb1-14a7-4133-9a89-297807894f69-kube-api-access-55cl9\" (UniqueName: \"kubernetes.io/projected/2b2c6fb1-14a7-4133-9a89-297807894f69-kube-api-access-55cl9\") pod \"2b2c6fb1-14a7-4133-9a89-297807894f69\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " Nov 02 07:56:54.375384 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.375362 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-metrics-certs\") pod \"2b2c6fb1-14a7-4133-9a89-297807894f69\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " Nov 02 07:56:54.375761 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.375740 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b2c6fb1-14a7-4133-9a89-297807894f69-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2c6fb1-14a7-4133-9a89-297807894f69-openshift-service-ca\") pod \"2b2c6fb1-14a7-4133-9a89-297807894f69\" (UID: \"2b2c6fb1-14a7-4133-9a89-297807894f69\") " Nov 02 07:56:54.376177 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.376136 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2c6fb1-14a7-4133-9a89-297807894f69-openshift-service-ca" pod "2b2c6fb1-14a7-4133-9a89-297807894f69" (UID: "2b2c6fb1-14a7-4133-9a89-297807894f69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:56:54.376367 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.376342 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-maas-api-tls" pod "2b2c6fb1-14a7-4133-9a89-297807894f69" (UID: "2b2c6fb1-14a7-4133-9a89-297807894f69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:56:54.376442 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.376379 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2c6fb1-14a7-4133-9a89-297807894f69-kube-api-access-55cl9" pod "2b2c6fb1-14a7-4133-9a89-297807894f69" (UID: "2b2c6fb1-14a7-4133-9a89-297807894f69"). InnerVolumeSpecName "kube-api-access-55cl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:56:54.377012 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.376993 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2c6fb1-14a7-4133-9a89-297807894f69-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:56:54.377123 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.377110 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:56:54.377230 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.377218 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55cl9\" (UniqueName: \"kubernetes.io/projected/2b2c6fb1-14a7-4133-9a89-297807894f69-kube-api-access-55cl9\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:56:54.378638 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.378613 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-metrics-certs" pod "2b2c6fb1-14a7-4133-9a89-297807894f69" (UID: "2b2c6fb1-14a7-4133-9a89-297807894f69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:56:54.478808 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.478777 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b2c6fb1-14a7-4133-9a89-297807894f69-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:56:54.814325 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.814228 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:54.961305 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.961268 5899 generic.go:356] "Generic (PLEG): container finished" podID="2b2c6fb1-14a7-4133-9a89-297807894f69" containerID="15fdb836f3b39a9379162c415efcb28f168fe628caf6b59fa23d31c45b66bb04" exitCode=0 Nov 02 07:56:54.961739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.961330 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" Nov 02 07:56:54.961739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.961346 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" event={"ID":"2b2c6fb1-14a7-4133-9a89-297807894f69","Type":"ContainerDied","Data":"15fdb836f3b39a9379162c415efcb28f168fe628caf6b59fa23d31c45b66bb04"} Nov 02 07:56:54.961739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.961403 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs" event={"ID":"2b2c6fb1-14a7-4133-9a89-297807894f69","Type":"ContainerDied","Data":"ff59eda2ce2d0165b5cf767b0294105b8163993bdb646e20bb9203ef349fd000"} Nov 02 07:56:54.961739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.961426 5899 scope.go:122] "RemoveContainer" containerID="15fdb836f3b39a9379162c415efcb28f168fe628caf6b59fa23d31c45b66bb04" Nov 02 07:56:54.972576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.972549 5899 scope.go:122] "RemoveContainer" containerID="15fdb836f3b39a9379162c415efcb28f168fe628caf6b59fa23d31c45b66bb04" Nov 02 07:56:54.972852 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:56:54.972834 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15fdb836f3b39a9379162c415efcb28f168fe628caf6b59fa23d31c45b66bb04\": container with ID starting with 15fdb836f3b39a9379162c415efcb28f168fe628caf6b59fa23d31c45b66bb04 not found: ID does not exist" containerID="15fdb836f3b39a9379162c415efcb28f168fe628caf6b59fa23d31c45b66bb04" Nov 02 07:56:54.972917 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:54.972860 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15fdb836f3b39a9379162c415efcb28f168fe628caf6b59fa23d31c45b66bb04"} err="failed to get container status \"15fdb836f3b39a9379162c415efcb28f168fe628caf6b59fa23d31c45b66bb04\": rpc error: code = NotFound desc = could not find container \"15fdb836f3b39a9379162c415efcb28f168fe628caf6b59fa23d31c45b66bb04\": container with ID starting with 15fdb836f3b39a9379162c415efcb28f168fe628caf6b59fa23d31c45b66bb04 not found: ID does not exist" Nov 02 07:56:55.013825 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:55.013790 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs"] Nov 02 07:56:55.018418 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:55.018388 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c7a27f-77c85f56d7-47vjs"] Nov 02 07:56:55.102368 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:55.102288 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:55.236870 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:55.236826 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:55.822582 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:55.822549 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2c6fb1-14a7-4133-9a89-297807894f69" path="/var/lib/kubelet/pods/2b2c6fb1-14a7-4133-9a89-297807894f69/volumes" Nov 02 07:56:55.875505 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:55.875454 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:56.249245 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:56.249213 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:56.842187 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:56.842145 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:57.013857 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:57.013823 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:58.674280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:58.674247 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:59.834921 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.834888 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf"] Nov 02 07:56:59.836482 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.836454 5899 cpu_manager.go:414] "containerMap: removing container" podUID="2b2c6fb1-14a7-4133-9a89-297807894f69" containerName="maas-api" Nov 02 07:56:59.836713 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.836696 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2c6fb1-14a7-4133-9a89-297807894f69" containerName="maas-api" Nov 02 07:56:59.837002 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.836983 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2c6fb1-14a7-4133-9a89-297807894f69" containerName="maas-api" Nov 02 07:56:59.842476 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.842455 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.846767 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.846732 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf"] Nov 02 07:56:59.869042 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.869013 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-6929a4\"/\"auth-test-model-6929a4-kserve-self-signed-certs\"" Nov 02 07:56:59.869208 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.869108 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-6929a4\"/\"default-dockercfg-bq4hx\"" Nov 02 07:56:59.869446 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.869415 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tmp-dir\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.869571 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.869556 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-dshm\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.869620 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.869603 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkksm\" (UniqueName: \"kubernetes.io/projected/42da3682-ca1a-468b-a4bb-6fe7f75d4279-kube-api-access-rkksm\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.869684 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.869667 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-6929a4\"/\"kube-root-ca.crt\"" Nov 02 07:56:59.869720 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.869688 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-model-cache\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.869760 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.869727 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-6929a4\"/\"openshift-service-ca.crt\"" Nov 02 07:56:59.869760 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.869740 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-home\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.869846 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.869807 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tls-certs\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.900993 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.900957 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:56:59.971796 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.971760 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tmp-dir\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.971992 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.971890 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-dshm\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.971992 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.971937 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkksm\" (UniqueName: \"kubernetes.io/projected/42da3682-ca1a-468b-a4bb-6fe7f75d4279-kube-api-access-rkksm\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.972131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.972018 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-model-cache\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.972131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.972088 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-home\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.972131 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.972117 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tls-certs\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.972339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.972226 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tmp-dir\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.972460 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.972435 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-model-cache\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.972630 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.972608 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-home\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.974227 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.974199 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-dshm\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.974869 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.974849 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tls-certs\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:56:59.995902 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:56:59.995871 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkksm\" (UniqueName: \"kubernetes.io/projected/42da3682-ca1a-468b-a4bb-6fe7f75d4279-kube-api-access-rkksm\") pod \"auth-test-model-6929a4-kserve-bd76b87b9-h4qhf\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:57:00.158566 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:00.158409 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:57:00.495153 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:00.495125 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf"] Nov 02 07:57:00.496186 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:57:00.496143 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42da3682_ca1a_468b_a4bb_6fe7f75d4279.slice/crio-0380d95cc81980c922f11a76388fbaacd651c4b1e2516fc8ce0507d5a07389de WatchSource:0}: Error finding container 0380d95cc81980c922f11a76388fbaacd651c4b1e2516fc8ce0507d5a07389de: Status 404 returned error can't find the container with id 0380d95cc81980c922f11a76388fbaacd651c4b1e2516fc8ce0507d5a07389de Nov 02 07:57:00.511030 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:00.510994 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:01.005339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:01.005304 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" event={"ID":"42da3682-ca1a-468b-a4bb-6fe7f75d4279","Type":"ContainerStarted","Data":"e1d5a843aea4c10b4f46173b1b205cb2065b261299bab6c4b18799f3481ada08"} Nov 02 07:57:01.005733 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:01.005355 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" event={"ID":"42da3682-ca1a-468b-a4bb-6fe7f75d4279","Type":"ContainerStarted","Data":"0380d95cc81980c922f11a76388fbaacd651c4b1e2516fc8ce0507d5a07389de"} Nov 02 07:57:01.046899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:01.046840 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" podStartSLOduration=2.046822743 podStartE2EDuration="2.046822743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:57:01.045207125 +0000 UTC m=+1921.343487950" watchObservedRunningTime="2026-11-02 07:57:01.046822743 +0000 UTC m=+1921.345103570" Nov 02 07:57:01.159608 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:01.159569 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:57:02.241922 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:02.241885 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:02.385810 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:02.385769 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:03.401912 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:03.401875 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:03.888521 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:03.888485 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2"] Nov 02 07:57:03.895509 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:03.891289 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" podUID="d44c845c-5941-41be-baa3-1baa0a2b559e" containerName="maas-api" containerID="cri-o://69d254510a3213ef1b437f30a66bab32cc9d6d98f3f90da0860f0fe03ecd98e6" gracePeriod=30 Nov 02 07:57:03.973592 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:03.973559 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:04.027118 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.027086 5899 generic.go:356] "Generic (PLEG): container finished" podID="d44c845c-5941-41be-baa3-1baa0a2b559e" containerID="69d254510a3213ef1b437f30a66bab32cc9d6d98f3f90da0860f0fe03ecd98e6" exitCode=0 Nov 02 07:57:04.027342 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.027146 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" event={"ID":"d44c845c-5941-41be-baa3-1baa0a2b559e","Type":"ContainerDied","Data":"69d254510a3213ef1b437f30a66bab32cc9d6d98f3f90da0860f0fe03ecd98e6"} Nov 02 07:57:04.259781 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.259742 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:57:04.352798 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.350586 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d44c845c-5941-41be-baa3-1baa0a2b559e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44c845c-5941-41be-baa3-1baa0a2b559e-openshift-service-ca\") pod \"d44c845c-5941-41be-baa3-1baa0a2b559e\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " Nov 02 07:57:04.352798 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.350680 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-maas-api-tls\") pod \"d44c845c-5941-41be-baa3-1baa0a2b559e\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " Nov 02 07:57:04.360628 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.357312 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44c845c-5941-41be-baa3-1baa0a2b559e-kube-api-access-zghcb\" (UniqueName: \"kubernetes.io/projected/d44c845c-5941-41be-baa3-1baa0a2b559e-kube-api-access-zghcb\") pod \"d44c845c-5941-41be-baa3-1baa0a2b559e\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " Nov 02 07:57:04.360628 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.358052 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-metrics-certs\") pod \"d44c845c-5941-41be-baa3-1baa0a2b559e\" (UID: \"d44c845c-5941-41be-baa3-1baa0a2b559e\") " Nov 02 07:57:04.364180 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.363854 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44c845c-5941-41be-baa3-1baa0a2b559e-openshift-service-ca" pod "d44c845c-5941-41be-baa3-1baa0a2b559e" (UID: "d44c845c-5941-41be-baa3-1baa0a2b559e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:57:04.395696 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.377251 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-metrics-certs" pod "d44c845c-5941-41be-baa3-1baa0a2b559e" (UID: "d44c845c-5941-41be-baa3-1baa0a2b559e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:57:04.441189 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.439339 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44c845c-5941-41be-baa3-1baa0a2b559e-kube-api-access-zghcb" pod "d44c845c-5941-41be-baa3-1baa0a2b559e" (UID: "d44c845c-5941-41be-baa3-1baa0a2b559e"). InnerVolumeSpecName "kube-api-access-zghcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:57:04.454185 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.452050 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-maas-api-tls" pod "d44c845c-5941-41be-baa3-1baa0a2b559e" (UID: "d44c845c-5941-41be-baa3-1baa0a2b559e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:57:04.467020 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.464371 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:57:04.467020 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.464411 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zghcb\" (UniqueName: \"kubernetes.io/projected/d44c845c-5941-41be-baa3-1baa0a2b559e-kube-api-access-zghcb\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:57:04.467020 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.464428 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d44c845c-5941-41be-baa3-1baa0a2b559e-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:57:04.467020 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.464446 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44c845c-5941-41be-baa3-1baa0a2b559e-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:57:04.504647 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.504609 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:04.803154 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:04.803117 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:05.034840 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:05.034802 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" event={"ID":"d44c845c-5941-41be-baa3-1baa0a2b559e","Type":"ContainerDied","Data":"8514c7b11a4c68fc91cd835878319ef83a0b9141eb5111b05008efcdd00ad246"} Nov 02 07:57:05.034840 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:05.034838 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2" Nov 02 07:57:05.035085 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:05.034863 5899 scope.go:122] "RemoveContainer" containerID="69d254510a3213ef1b437f30a66bab32cc9d6d98f3f90da0860f0fe03ecd98e6" Nov 02 07:57:05.093325 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:05.093261 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2"] Nov 02 07:57:05.102828 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:05.102803 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-680c01-7c6df8b856-z2zn2"] Nov 02 07:57:05.820339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:05.820310 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d44c845c-5941-41be-baa3-1baa0a2b559e" path="/var/lib/kubelet/pods/d44c845c-5941-41be-baa3-1baa0a2b559e/volumes" Nov 02 07:57:06.502732 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:06.502702 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:08.226971 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:08.226922 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:09.778586 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:09.778553 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:10.067411 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:10.067320 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:10.159193 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:10.159133 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:57:10.172843 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:10.172817 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:57:10.244402 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:10.244345 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:10.570692 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:10.570658 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:10.946466 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:10.946368 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:11.112854 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:11.112825 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 07:57:11.860458 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:11.860418 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:12.196498 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:12.196401 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:12.336136 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:12.336095 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:12.495244 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:12.495209 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:13.145007 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:57:13.144971 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:57:14.002765 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:14.002726 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:15.945293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:15.945260 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:16.666110 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:16.666078 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:16.810393 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:16.810342 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:17.909583 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:17.909527 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:18.363923 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:18.363887 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:18.507339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:18.507303 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:19.794788 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:19.794755 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:21.549263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:21.549228 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:23.225506 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:23.225465 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:24.879597 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:24.879557 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:26.582488 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:26.582454 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:28.380171 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:28.380119 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:30.207028 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:30.206993 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:31.923791 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:31.923755 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:33.654831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:33.654797 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:35.397688 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:35.397651 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:37.084506 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:37.084465 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:38.791582 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:38.791543 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:40.378434 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.378399 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb"] Nov 02 07:57:40.379006 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.378992 5899 cpu_manager.go:414] "containerMap: removing container" podUID="d44c845c-5941-41be-baa3-1baa0a2b559e" containerName="maas-api" Nov 02 07:57:40.379053 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.379007 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44c845c-5941-41be-baa3-1baa0a2b559e" containerName="maas-api" Nov 02 07:57:40.379130 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.379121 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44c845c-5941-41be-baa3-1baa0a2b559e" containerName="maas-api" Nov 02 07:57:40.383090 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.383071 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:40.414541 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.414506 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-ac716b-serving-cert\"" Nov 02 07:57:40.417488 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.417455 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb"] Nov 02 07:57:40.481870 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.481832 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-openshift-service-ca\") pod \"maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:40.482037 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.481901 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj5wv\" (UniqueName: \"kubernetes.io/projected/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-kube-api-access-bj5wv\") pod \"maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:40.482093 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.482073 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-metrics-certs\") pod \"maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:40.482140 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.482128 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-maas-api-tls\") pod \"maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:40.510460 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.510424 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:40.583385 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.583350 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-openshift-service-ca\") pod \"maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:40.583589 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.583427 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5wv\" (UniqueName: \"kubernetes.io/projected/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-kube-api-access-bj5wv\") pod \"maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:40.583654 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.583597 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-metrics-certs\") pod \"maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:40.583716 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.583668 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-maas-api-tls\") pod \"maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:40.584498 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.584405 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-openshift-service-ca\") pod \"maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:40.586532 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.586509 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-maas-api-tls\") pod \"maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:40.586990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.586952 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-metrics-certs\") pod \"maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:40.616124 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.616053 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5wv\" (UniqueName: \"kubernetes.io/projected/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-kube-api-access-bj5wv\") pod \"maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:40.654731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.654630 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:40.697258 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:40.697221 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:41.010444 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:41.010415 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb"] Nov 02 07:57:41.012751 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:57:41.012724 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0406f5ad_ef2c_4cb0_b8dd_cb3d98d717b3.slice/crio-8ab67ae63213356a4b6a01b91a1e5273fdeecb9c7223a899aaf92728bed63006 WatchSource:0}: Error finding container 8ab67ae63213356a4b6a01b91a1e5273fdeecb9c7223a899aaf92728bed63006: Status 404 returned error can't find the container with id 8ab67ae63213356a4b6a01b91a1e5273fdeecb9c7223a899aaf92728bed63006 Nov 02 07:57:41.280449 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:41.280346 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" event={"ID":"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3","Type":"ContainerStarted","Data":"8ab67ae63213356a4b6a01b91a1e5273fdeecb9c7223a899aaf92728bed63006"} Nov 02 07:57:42.288435 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:42.288402 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" event={"ID":"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3","Type":"ContainerStarted","Data":"b4b1df4e3f9681c933660e41340a240f9836e9d7f9073844a5fcf97593b6e7f3"} Nov 02 07:57:42.288815 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:42.288475 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:42.337198 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:42.337112 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" podStartSLOduration=1.968725127 podStartE2EDuration="2.33709614s" totalImagesPullingTime="368.371013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:57:42.333518454 +0000 UTC m=+1962.631799279" watchObservedRunningTime="2026-11-02 07:57:42.33709614 +0000 UTC m=+1962.635376966" Nov 02 07:57:42.725614 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:42.725531 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:43.140429 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:57:43.140389 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:57:44.501152 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:44.501113 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:46.199394 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:46.199360 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:47.908264 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:47.908229 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:48.299912 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:48.299883 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:57:49.614372 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:49.614334 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:51.407596 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:51.407559 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:53.185503 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:53.185470 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:53.629929 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:53.629896 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:54.926876 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:54.926842 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:56.682463 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:56.682405 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:57:58.390486 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:57:58.390449 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:00.148277 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:00.148242 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:02.174629 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:02.174594 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:04.015918 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:04.015868 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:04.634292 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:04.634253 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:05.899685 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:05.899637 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:07.613510 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:07.613452 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:09.379581 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:09.379546 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:11.232395 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:11.232302 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:12.627719 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:12.627682 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:12.982439 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:12.982407 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:13.145350 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:58:13.145310 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:58:14.246294 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:14.246258 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:14.825472 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:14.825441 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:16.537862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.537826 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx"] Nov 02 07:58:16.538357 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.538035 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" podUID="e7d2a632-5b70-4753-9ea0-090a7a41bbd8" containerName="discovery" containerID="cri-o://b29f1bd22c274502bdbd3585e8143b3ef56e1f48995262102781a60b1ba58119" gracePeriod=30 Nov 02 07:58:16.701925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.701887 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:16.838875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.838844 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:58:16.887817 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.887784 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-local-certs\") pod \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " Nov 02 07:58:16.888004 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.887837 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-kube-api-access-8v4w8\" (UniqueName: \"kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-kube-api-access-8v4w8\") pod \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " Nov 02 07:58:16.888004 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.887886 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-ca-configmap\") pod \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " Nov 02 07:58:16.888004 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.887970 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-token\" (UniqueName: \"kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-token\") pod \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " Nov 02 07:58:16.888227 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.888082 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-kubeconfig\") pod \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " Nov 02 07:58:16.888227 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.888111 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-cacerts\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-cacerts\") pod \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " Nov 02 07:58:16.888227 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.888214 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-dns-cert\") pod \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\" (UID: \"e7d2a632-5b70-4753-9ea0-090a7a41bbd8\") " Nov 02 07:58:16.888500 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.888471 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-ca-configmap" pod "e7d2a632-5b70-4753-9ea0-090a7a41bbd8" (UID: "e7d2a632-5b70-4753-9ea0-090a7a41bbd8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:58:16.888811 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.888788 5899 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-ca-configmap\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:16.891252 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.891219 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-cacerts" pod "e7d2a632-5b70-4753-9ea0-090a7a41bbd8" (UID: "e7d2a632-5b70-4753-9ea0-090a7a41bbd8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:58:16.891252 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.891244 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-dns-cert" pod "e7d2a632-5b70-4753-9ea0-090a7a41bbd8" (UID: "e7d2a632-5b70-4753-9ea0-090a7a41bbd8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:58:16.891427 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.891377 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-kube-api-access-8v4w8" pod "e7d2a632-5b70-4753-9ea0-090a7a41bbd8" (UID: "e7d2a632-5b70-4753-9ea0-090a7a41bbd8"). InnerVolumeSpecName "kube-api-access-8v4w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:58:16.891487 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.891418 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-local-certs" pod "e7d2a632-5b70-4753-9ea0-090a7a41bbd8" (UID: "e7d2a632-5b70-4753-9ea0-090a7a41bbd8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:58:16.892734 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.892711 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-token" pod "e7d2a632-5b70-4753-9ea0-090a7a41bbd8" (UID: "e7d2a632-5b70-4753-9ea0-090a7a41bbd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:58:16.892838 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.892713 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-kubeconfig" pod "e7d2a632-5b70-4753-9ea0-090a7a41bbd8" (UID: "e7d2a632-5b70-4753-9ea0-090a7a41bbd8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:58:16.990494 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.990460 5899 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-kubeconfig\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:16.990494 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.990490 5899 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-cacerts\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:16.990494 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.990502 5899 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-csr-dns-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:16.990742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.990511 5899 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-local-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:16.990742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.990520 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v4w8\" (UniqueName: \"kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-kube-api-access-8v4w8\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:16.990742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:16.990530 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d2a632-5b70-4753-9ea0-090a7a41bbd8-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:17.512622 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:17.512589 5899 generic.go:356] "Generic (PLEG): container finished" podID="e7d2a632-5b70-4753-9ea0-090a7a41bbd8" containerID="b29f1bd22c274502bdbd3585e8143b3ef56e1f48995262102781a60b1ba58119" exitCode=0 Nov 02 07:58:17.512835 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:17.512639 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" event={"ID":"e7d2a632-5b70-4753-9ea0-090a7a41bbd8","Type":"ContainerDied","Data":"b29f1bd22c274502bdbd3585e8143b3ef56e1f48995262102781a60b1ba58119"} Nov 02 07:58:17.512835 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:17.512642 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" Nov 02 07:58:17.512835 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:17.512679 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx" event={"ID":"e7d2a632-5b70-4753-9ea0-090a7a41bbd8","Type":"ContainerDied","Data":"98c165bf848222dea8c83072a99ede789f07b5a753fc9c645ba83a5880048db4"} Nov 02 07:58:17.512835 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:17.512701 5899 scope.go:122] "RemoveContainer" containerID="b29f1bd22c274502bdbd3585e8143b3ef56e1f48995262102781a60b1ba58119" Nov 02 07:58:17.524334 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:17.524314 5899 scope.go:122] "RemoveContainer" containerID="b29f1bd22c274502bdbd3585e8143b3ef56e1f48995262102781a60b1ba58119" Nov 02 07:58:17.524626 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:58:17.524607 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b29f1bd22c274502bdbd3585e8143b3ef56e1f48995262102781a60b1ba58119\": container with ID starting with b29f1bd22c274502bdbd3585e8143b3ef56e1f48995262102781a60b1ba58119 not found: ID does not exist" containerID="b29f1bd22c274502bdbd3585e8143b3ef56e1f48995262102781a60b1ba58119" Nov 02 07:58:17.524697 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:17.524634 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b29f1bd22c274502bdbd3585e8143b3ef56e1f48995262102781a60b1ba58119"} err="failed to get container status \"b29f1bd22c274502bdbd3585e8143b3ef56e1f48995262102781a60b1ba58119\": rpc error: code = NotFound desc = could not find container \"b29f1bd22c274502bdbd3585e8143b3ef56e1f48995262102781a60b1ba58119\": container with ID starting with b29f1bd22c274502bdbd3585e8143b3ef56e1f48995262102781a60b1ba58119 not found: ID does not exist" Nov 02 07:58:17.578990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:17.578959 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx"] Nov 02 07:58:17.582826 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:17.582776 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ftdzx"] Nov 02 07:58:17.821222 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:17.821122 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7d2a632-5b70-4753-9ea0-090a7a41bbd8" path="/var/lib/kubelet/pods/e7d2a632-5b70-4753-9ea0-090a7a41bbd8/volumes" Nov 02 07:58:18.510057 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:18.510022 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:20.193872 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:20.193836 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:21.629460 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:21.629422 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7"] Nov 02 07:58:21.629989 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:21.629613 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" podUID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" containerName="istio-proxy" containerID="cri-o://356b48937f9318be9aa0388bf6388807344fcdf085c26f3aac3248e42ed3f895" gracePeriod=30 Nov 02 07:58:21.640581 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:21.640248 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc"] Nov 02 07:58:21.640581 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:21.640476 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" podUID="9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" containerName="istio-proxy" containerID="cri-o://ea83ba6df2c7237f1aa122d163d80c5901b597a1db819994d874c2ea42e04cf0" gracePeriod=30 Nov 02 07:58:21.923138 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:21.923055 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:23.706733 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:23.706697 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:25.503995 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:25.503954 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:26.376210 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:26.376169 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-d2sj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 07:58:26.376407 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:26.376226 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" podUID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 07:58:26.920815 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:26.920749 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:58:26.953636 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:26.953606 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:58:27.021147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021115 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-certs\") pod \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " Nov 02 07:58:27.021147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021151 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-socket\") pod \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " Nov 02 07:58:27.021412 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021202 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-kube-api-access-wbldj\" (UniqueName: \"kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-kube-api-access-wbldj\") pod \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " Nov 02 07:58:27.021412 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021239 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-podinfo\") pod \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " Nov 02 07:58:27.021412 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021270 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-data\") pod \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " Nov 02 07:58:27.021412 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021348 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-token\" (UniqueName: \"kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-token\") pod \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " Nov 02 07:58:27.021412 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021402 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-credential-socket\") pod \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " Nov 02 07:58:27.021673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021427 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istiod-ca-cert\") pod \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " Nov 02 07:58:27.021673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021468 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-podinfo\") pod \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " Nov 02 07:58:27.021673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021475 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-certs" pod "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" (UID: "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:58:27.021673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021496 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-envoy\") pod \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " Nov 02 07:58:27.021673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021510 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-socket" pod "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" (UID: "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:58:27.021673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021610 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-envoy\") pod \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " Nov 02 07:58:27.021673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021617 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-data" pod "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" (UID: "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:58:27.021673 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021674 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-kube-api-access-zwxpw\" (UniqueName: \"kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-kube-api-access-zwxpw\") pod \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " Nov 02 07:58:27.022075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021703 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-certs\") pod \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " Nov 02 07:58:27.022075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021739 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-data\") pod \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " Nov 02 07:58:27.022075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021832 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-socket\") pod \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " Nov 02 07:58:27.022075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021901 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-token\" (UniqueName: \"kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-token\") pod \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " Nov 02 07:58:27.022075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.021920 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istiod-ca-cert" pod "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" (UID: "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:58:27.022075 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.022029 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-credential-socket\") pod \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\" (UID: \"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5\") " Nov 02 07:58:27.022424 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.022101 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istiod-ca-cert\") pod \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\" (UID: \"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5\") " Nov 02 07:58:27.022424 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.022174 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-credential-socket" pod "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" (UID: "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:58:27.022546 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.022423 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-credential-socket" pod "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" (UID: "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:58:27.022749 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.022724 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istiod-ca-cert" pod "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" (UID: "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:58:27.023497 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.022916 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.023497 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.022942 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.023497 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.022957 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.023497 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.022971 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.023497 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.022984 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.023497 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.022997 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.024321 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.024242 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-socket" pod "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" (UID: "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:58:27.024321 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.024262 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-data" pod "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" (UID: "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:58:27.024321 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.024284 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-certs" pod "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" (UID: "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:58:27.024524 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.024446 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-token" pod "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" (UID: "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:58:27.025106 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.025081 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-envoy" pod "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" (UID: "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:58:27.025106 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.025089 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-kube-api-access-wbldj" pod "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" (UID: "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5"). InnerVolumeSpecName "kube-api-access-wbldj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:58:27.025332 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.025112 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-token" pod "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" (UID: "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:58:27.025400 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.025353 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-envoy" pod "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" (UID: "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:58:27.025779 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.025750 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-kube-api-access-zwxpw" pod "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" (UID: "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5"). InnerVolumeSpecName "kube-api-access-zwxpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:58:27.026127 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.026106 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-podinfo" pod "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" (UID: "9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:58:27.026410 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.026392 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-podinfo" pod "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" (UID: "3498375e-3b5a-4be9-92b1-dcfb2ebd76c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:58:27.124703 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.124669 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.124703 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.124699 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.124703 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.124707 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.124925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.124715 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.124925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.124724 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwxpw\" (UniqueName: \"kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-kube-api-access-zwxpw\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.124925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.124734 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.124925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.124744 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.124925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.124751 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.124925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.124760 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.124925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.124767 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.124925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.124775 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbldj\" (UniqueName: \"kubernetes.io/projected/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-kube-api-access-wbldj\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.124925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.124782 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:58:27.274888 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.274854 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:27.593739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.593647 5899 generic.go:356] "Generic (PLEG): container finished" podID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" containerID="356b48937f9318be9aa0388bf6388807344fcdf085c26f3aac3248e42ed3f895" exitCode=0 Nov 02 07:58:27.593739 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.593710 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" Nov 02 07:58:27.593973 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.593730 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" event={"ID":"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5","Type":"ContainerDied","Data":"356b48937f9318be9aa0388bf6388807344fcdf085c26f3aac3248e42ed3f895"} Nov 02 07:58:27.593973 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.593805 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7" event={"ID":"3498375e-3b5a-4be9-92b1-dcfb2ebd76c5","Type":"ContainerDied","Data":"eb8d6f1802a49254e93845f25e9e9340414890668510cfa619287968d47c5acd"} Nov 02 07:58:27.593973 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.593831 5899 scope.go:122] "RemoveContainer" containerID="356b48937f9318be9aa0388bf6388807344fcdf085c26f3aac3248e42ed3f895" Nov 02 07:58:27.595430 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.595404 5899 generic.go:356] "Generic (PLEG): container finished" podID="9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" containerID="ea83ba6df2c7237f1aa122d163d80c5901b597a1db819994d874c2ea42e04cf0" exitCode=0 Nov 02 07:58:27.595564 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.595461 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" event={"ID":"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5","Type":"ContainerDied","Data":"ea83ba6df2c7237f1aa122d163d80c5901b597a1db819994d874c2ea42e04cf0"} Nov 02 07:58:27.595564 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.595472 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" Nov 02 07:58:27.595564 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.595500 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc" event={"ID":"9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5","Type":"ContainerDied","Data":"a37f722b1e783b246459859c5e3b63d9c606593ce3a030449a1fb03f9f7aae0a"} Nov 02 07:58:27.607371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.607354 5899 scope.go:122] "RemoveContainer" containerID="356b48937f9318be9aa0388bf6388807344fcdf085c26f3aac3248e42ed3f895" Nov 02 07:58:27.607673 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:58:27.607649 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"356b48937f9318be9aa0388bf6388807344fcdf085c26f3aac3248e42ed3f895\": container with ID starting with 356b48937f9318be9aa0388bf6388807344fcdf085c26f3aac3248e42ed3f895 not found: ID does not exist" containerID="356b48937f9318be9aa0388bf6388807344fcdf085c26f3aac3248e42ed3f895" Nov 02 07:58:27.607734 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.607687 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356b48937f9318be9aa0388bf6388807344fcdf085c26f3aac3248e42ed3f895"} err="failed to get container status \"356b48937f9318be9aa0388bf6388807344fcdf085c26f3aac3248e42ed3f895\": rpc error: code = NotFound desc = could not find container \"356b48937f9318be9aa0388bf6388807344fcdf085c26f3aac3248e42ed3f895\": container with ID starting with 356b48937f9318be9aa0388bf6388807344fcdf085c26f3aac3248e42ed3f895 not found: ID does not exist" Nov 02 07:58:27.607734 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.607708 5899 scope.go:122] "RemoveContainer" containerID="ea83ba6df2c7237f1aa122d163d80c5901b597a1db819994d874c2ea42e04cf0" Nov 02 07:58:27.617117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.617097 5899 scope.go:122] "RemoveContainer" containerID="ea83ba6df2c7237f1aa122d163d80c5901b597a1db819994d874c2ea42e04cf0" Nov 02 07:58:27.617415 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:58:27.617396 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea83ba6df2c7237f1aa122d163d80c5901b597a1db819994d874c2ea42e04cf0\": container with ID starting with ea83ba6df2c7237f1aa122d163d80c5901b597a1db819994d874c2ea42e04cf0 not found: ID does not exist" containerID="ea83ba6df2c7237f1aa122d163d80c5901b597a1db819994d874c2ea42e04cf0" Nov 02 07:58:27.617486 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.617426 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea83ba6df2c7237f1aa122d163d80c5901b597a1db819994d874c2ea42e04cf0"} err="failed to get container status \"ea83ba6df2c7237f1aa122d163d80c5901b597a1db819994d874c2ea42e04cf0\": rpc error: code = NotFound desc = could not find container \"ea83ba6df2c7237f1aa122d163d80c5901b597a1db819994d874c2ea42e04cf0\": container with ID starting with ea83ba6df2c7237f1aa122d163d80c5901b597a1db819994d874c2ea42e04cf0 not found: ID does not exist" Nov 02 07:58:27.645698 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.645675 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7"] Nov 02 07:58:27.657276 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.657250 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-d2sj7"] Nov 02 07:58:27.690211 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.690147 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc"] Nov 02 07:58:27.701925 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.701898 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cljsc"] Nov 02 07:58:27.820518 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.820485 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" path="/var/lib/kubelet/pods/3498375e-3b5a-4be9-92b1-dcfb2ebd76c5/volumes" Nov 02 07:58:27.820968 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:27.820955 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" path="/var/lib/kubelet/pods/9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5/volumes" Nov 02 07:58:29.062147 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:29.062119 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:30.896896 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:30.896863 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:32.606349 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:32.606295 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:34.331534 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:34.331500 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:36.148753 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:36.148710 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:37.920268 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:37.920232 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:39.009444 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:39.009400 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:39.466849 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:39.466762 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:39.704178 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:39.704133 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:41.652175 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:41.652123 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:43.141146 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:58:43.141108 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:58:43.448740 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:43.448661 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:44.490380 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:44.490348 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:45.219558 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:45.219527 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:47.011933 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:47.011887 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:48.767455 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:48.767423 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:49.607489 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:49.607441 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:50.702092 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:50.702058 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:52.513339 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:52.513296 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:54.346759 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.346722 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:54.491599 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.491561 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:54.534561 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.534521 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7361-149f84"] Nov 02 07:58:54.535418 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.535394 5899 cpu_manager.go:414] "containerMap: removing container" podUID="9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" containerName="istio-proxy" Nov 02 07:58:54.535533 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.535419 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" containerName="istio-proxy" Nov 02 07:58:54.535533 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.535476 5899 cpu_manager.go:414] "containerMap: removing container" podUID="e7d2a632-5b70-4753-9ea0-090a7a41bbd8" containerName="discovery" Nov 02 07:58:54.535533 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.535484 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d2a632-5b70-4753-9ea0-090a7a41bbd8" containerName="discovery" Nov 02 07:58:54.535533 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.535506 5899 cpu_manager.go:414] "containerMap: removing container" podUID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" containerName="istio-proxy" Nov 02 07:58:54.535533 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.535514 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" containerName="istio-proxy" Nov 02 07:58:54.535724 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.535665 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e41c1b1-9b63-4bc3-b199-0d174a0cc8a5" containerName="istio-proxy" Nov 02 07:58:54.535724 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.535684 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d2a632-5b70-4753-9ea0-090a7a41bbd8" containerName="discovery" Nov 02 07:58:54.535724 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.535694 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="3498375e-3b5a-4be9-92b1-dcfb2ebd76c5" containerName="istio-proxy" Nov 02 07:58:54.539708 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.539681 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-149f84" Nov 02 07:58:54.553699 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.552633 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-149f84"] Nov 02 07:58:54.564434 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.564403 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pmvw6\"" Nov 02 07:58:54.718832 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.718741 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvl6p\" (UniqueName: \"kubernetes.io/projected/1983efdc-6a25-4e29-8a58-735284644376-kube-api-access-mvl6p\") pod \"test-curl-7361-149f84\" (UID: \"1983efdc-6a25-4e29-8a58-735284644376\") " pod="opendatahub/test-curl-7361-149f84" Nov 02 07:58:54.819521 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.819487 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvl6p\" (UniqueName: \"kubernetes.io/projected/1983efdc-6a25-4e29-8a58-735284644376-kube-api-access-mvl6p\") pod \"test-curl-7361-149f84\" (UID: \"1983efdc-6a25-4e29-8a58-735284644376\") " pod="opendatahub/test-curl-7361-149f84" Nov 02 07:58:54.842176 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.842130 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvl6p\" (UniqueName: \"kubernetes.io/projected/1983efdc-6a25-4e29-8a58-735284644376-kube-api-access-mvl6p\") pod \"test-curl-7361-149f84\" (UID: \"1983efdc-6a25-4e29-8a58-735284644376\") " pod="opendatahub/test-curl-7361-149f84" Nov 02 07:58:54.859246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:54.859210 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-149f84" Nov 02 07:58:55.157459 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:55.157432 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-149f84"] Nov 02 07:58:55.159190 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:58:55.159139 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1983efdc_6a25_4e29_8a58_735284644376.slice/crio-ab284fbcce7c7d0d6bcc8efad85a8ea2dc5109d49de8e69964b1eca36c0ec15a WatchSource:0}: Error finding container ab284fbcce7c7d0d6bcc8efad85a8ea2dc5109d49de8e69964b1eca36c0ec15a: Status 404 returned error can't find the container with id ab284fbcce7c7d0d6bcc8efad85a8ea2dc5109d49de8e69964b1eca36c0ec15a Nov 02 07:58:55.780454 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:55.780414 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-149f84" event={"ID":"1983efdc-6a25-4e29-8a58-735284644376","Type":"ContainerStarted","Data":"ab284fbcce7c7d0d6bcc8efad85a8ea2dc5109d49de8e69964b1eca36c0ec15a"} Nov 02 07:58:56.308108 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:56.308069 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:57.796212 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:57.796138 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-149f84" event={"ID":"1983efdc-6a25-4e29-8a58-735284644376","Type":"ContainerStarted","Data":"d254e2cbb63f609e0ae067ccf452978db39bd9283653ee4426266ea4fe13a534"} Nov 02 07:58:57.833842 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:57.833790 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7361-149f84" podStartSLOduration=1.719628814 podStartE2EDuration="3.833777143s" totalImagesPullingTime="2.114148329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:58:57.833525776 +0000 UTC m=+2038.131806602" watchObservedRunningTime="2026-11-02 07:58:57.833777143 +0000 UTC m=+2038.132057968" Nov 02 07:58:58.106637 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:58.106539 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:58:58.687862 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:58.687829 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7361-149f84"] Nov 02 07:58:58.692236 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:58.692204 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7361-149f84"] Nov 02 07:58:58.835221 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:58.835189 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7361-22fe8c"] Nov 02 07:58:58.835861 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:58.835841 5899 cpu_manager.go:414] "containerMap: removing container" podUID="1983efdc-6a25-4e29-8a58-735284644376" containerName="test-curl-7361-149f84" Nov 02 07:58:58.835903 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:58.835863 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1983efdc-6a25-4e29-8a58-735284644376" containerName="test-curl-7361-149f84" Nov 02 07:58:58.835976 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:58.835965 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="1983efdc-6a25-4e29-8a58-735284644376" containerName="test-curl-7361-149f84" Nov 02 07:58:58.839469 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:58.839453 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-22fe8c" Nov 02 07:58:58.854847 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:58.854818 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-22fe8c"] Nov 02 07:58:58.873990 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:58.873959 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb29q\" (UniqueName: \"kubernetes.io/projected/97370d71-3b32-4d01-8130-3bff59c4d1b0-kube-api-access-jb29q\") pod \"test-curl-7361-22fe8c\" (UID: \"97370d71-3b32-4d01-8130-3bff59c4d1b0\") " pod="opendatahub/test-curl-7361-22fe8c" Nov 02 07:58:58.975441 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:58.975359 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb29q\" (UniqueName: \"kubernetes.io/projected/97370d71-3b32-4d01-8130-3bff59c4d1b0-kube-api-access-jb29q\") pod \"test-curl-7361-22fe8c\" (UID: \"97370d71-3b32-4d01-8130-3bff59c4d1b0\") " pod="opendatahub/test-curl-7361-22fe8c" Nov 02 07:58:58.998476 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:58.998451 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb29q\" (UniqueName: \"kubernetes.io/projected/97370d71-3b32-4d01-8130-3bff59c4d1b0-kube-api-access-jb29q\") pod \"test-curl-7361-22fe8c\" (UID: \"97370d71-3b32-4d01-8130-3bff59c4d1b0\") " pod="opendatahub/test-curl-7361-22fe8c" Nov 02 07:58:59.152241 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:59.152207 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-22fe8c" Nov 02 07:58:59.457156 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:59.457131 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-22fe8c"] Nov 02 07:58:59.458256 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:58:59.458232 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97370d71_3b32_4d01_8130_3bff59c4d1b0.slice/crio-9fe9d828b542cf20371bae94b607296ec19c0c7f88ccf17efa5e83bc87300db1 WatchSource:0}: Error finding container 9fe9d828b542cf20371bae94b607296ec19c0c7f88ccf17efa5e83bc87300db1: Status 404 returned error can't find the container with id 9fe9d828b542cf20371bae94b607296ec19c0c7f88ccf17efa5e83bc87300db1 Nov 02 07:58:59.810420 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:59.810384 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-22fe8c" event={"ID":"97370d71-3b32-4d01-8130-3bff59c4d1b0","Type":"ContainerStarted","Data":"1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489"} Nov 02 07:58:59.810420 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:59.810427 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-22fe8c" event={"ID":"97370d71-3b32-4d01-8130-3bff59c4d1b0","Type":"ContainerStarted","Data":"9fe9d828b542cf20371bae94b607296ec19c0c7f88ccf17efa5e83bc87300db1"} Nov 02 07:58:59.810684 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:59.810577 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7361-149f84" podUID="1983efdc-6a25-4e29-8a58-735284644376" containerName="test-curl-7361-149f84" containerID="cri-o://d254e2cbb63f609e0ae067ccf452978db39bd9283653ee4426266ea4fe13a534" gracePeriod=2 Nov 02 07:58:59.847646 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:59.847591 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7361-22fe8c" podStartSLOduration=1.847574363 podStartE2EDuration="1.847574363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:58:59.847177705 +0000 UTC m=+2040.145458528" watchObservedRunningTime="2026-11-02 07:58:59.847574363 +0000 UTC m=+2040.145855191" Nov 02 07:58:59.890270 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:58:59.890229 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:00.695859 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:00.695545 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7361-22fe8c"] Nov 02 07:59:00.701659 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:00.701627 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7361-22fe8c"] Nov 02 07:59:00.913858 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:00.913827 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7361-f96a04"] Nov 02 07:59:00.914477 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:00.914462 5899 cpu_manager.go:414] "containerMap: removing container" podUID="97370d71-3b32-4d01-8130-3bff59c4d1b0" containerName="test-curl-7361-22fe8c" Nov 02 07:59:00.914534 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:00.914479 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97370d71-3b32-4d01-8130-3bff59c4d1b0" containerName="test-curl-7361-22fe8c" Nov 02 07:59:00.914588 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:00.914578 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="97370d71-3b32-4d01-8130-3bff59c4d1b0" containerName="test-curl-7361-22fe8c" Nov 02 07:59:00.919451 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:00.919434 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-f96a04" Nov 02 07:59:00.934669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:00.934639 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-f96a04"] Nov 02 07:59:01.002722 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:01.002689 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm5fk\" (UniqueName: \"kubernetes.io/projected/66d0f1d8-5ff1-4469-9539-51eb8d91fbaf-kube-api-access-xm5fk\") pod \"test-curl-7361-f96a04\" (UID: \"66d0f1d8-5ff1-4469-9539-51eb8d91fbaf\") " pod="opendatahub/test-curl-7361-f96a04" Nov 02 07:59:01.104501 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:01.104440 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm5fk\" (UniqueName: \"kubernetes.io/projected/66d0f1d8-5ff1-4469-9539-51eb8d91fbaf-kube-api-access-xm5fk\") pod \"test-curl-7361-f96a04\" (UID: \"66d0f1d8-5ff1-4469-9539-51eb8d91fbaf\") " pod="opendatahub/test-curl-7361-f96a04" Nov 02 07:59:01.127284 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:01.127247 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm5fk\" (UniqueName: \"kubernetes.io/projected/66d0f1d8-5ff1-4469-9539-51eb8d91fbaf-kube-api-access-xm5fk\") pod \"test-curl-7361-f96a04\" (UID: \"66d0f1d8-5ff1-4469-9539-51eb8d91fbaf\") " pod="opendatahub/test-curl-7361-f96a04" Nov 02 07:59:01.231746 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:01.231710 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-f96a04" Nov 02 07:59:01.530483 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:01.530453 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-f96a04"] Nov 02 07:59:01.672301 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:01.672262 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:01.814914 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:01.814881 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:01.825899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:01.825871 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-f96a04" event={"ID":"66d0f1d8-5ff1-4469-9539-51eb8d91fbaf","Type":"ContainerStarted","Data":"b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09"} Nov 02 07:59:01.826025 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:01.825890 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7361-22fe8c" podUID="97370d71-3b32-4d01-8130-3bff59c4d1b0" containerName="test-curl-7361-22fe8c" containerID="cri-o://1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489" gracePeriod=2 Nov 02 07:59:01.826025 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:01.825913 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-f96a04" event={"ID":"66d0f1d8-5ff1-4469-9539-51eb8d91fbaf","Type":"ContainerStarted","Data":"375106be7ab934208a6524930b2348299001738354740a0a09b452af96ab28fd"} Nov 02 07:59:01.868255 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:01.868214 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7361-f96a04" podStartSLOduration=1.86819763 podStartE2EDuration="1.86819763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:01.865102519 +0000 UTC m=+2042.163383344" watchObservedRunningTime="2026-11-02 07:59:01.86819763 +0000 UTC m=+2042.166478454" Nov 02 07:59:02.003580 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.003552 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-149f84" Nov 02 07:59:02.017016 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:02.016978 5899 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-149f84\" is forbidden: User \"system:node:ip-10-0-135-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-6.ec2.internal' and this object" podUID="1983efdc-6a25-4e29-8a58-735284644376" pod="opendatahub/test-curl-7361-149f84" Nov 02 07:59:02.116524 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.116442 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1983efdc-6a25-4e29-8a58-735284644376-kube-api-access-mvl6p\" (UniqueName: \"kubernetes.io/projected/1983efdc-6a25-4e29-8a58-735284644376-kube-api-access-mvl6p\") pod \"1983efdc-6a25-4e29-8a58-735284644376\" (UID: \"1983efdc-6a25-4e29-8a58-735284644376\") " Nov 02 07:59:02.118420 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.118396 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1983efdc-6a25-4e29-8a58-735284644376-kube-api-access-mvl6p" pod "1983efdc-6a25-4e29-8a58-735284644376" (UID: "1983efdc-6a25-4e29-8a58-735284644376"). InnerVolumeSpecName "kube-api-access-mvl6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:59:02.219088 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.219049 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvl6p\" (UniqueName: \"kubernetes.io/projected/1983efdc-6a25-4e29-8a58-735284644376-kube-api-access-mvl6p\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:02.646979 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.646942 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg"] Nov 02 07:59:02.647213 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.647150 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" podUID="c1f03252-734a-44ad-9c2a-4072303b9a09" containerName="istio-proxy" containerID="cri-o://5f036726866260a4f0aa7f2c289ba6b46b09b5b5f37ae37f576aa786948e35d1" gracePeriod=30 Nov 02 07:59:02.674756 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.674726 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7361-f96a04"] Nov 02 07:59:02.681399 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.681368 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7361-f96a04"] Nov 02 07:59:02.704597 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:02.704561 5899 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-149f84\" is forbidden: User \"system:node:ip-10-0-135-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-6.ec2.internal' and this object" podUID="1983efdc-6a25-4e29-8a58-735284644376" pod="opendatahub/test-curl-7361-149f84" Nov 02 07:59:02.828831 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.828798 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7361-901598"] Nov 02 07:59:02.829390 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.829375 5899 cpu_manager.go:414] "containerMap: removing container" podUID="66d0f1d8-5ff1-4469-9539-51eb8d91fbaf" containerName="test-curl-7361-f96a04" Nov 02 07:59:02.829503 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.829391 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d0f1d8-5ff1-4469-9539-51eb8d91fbaf" containerName="test-curl-7361-f96a04" Nov 02 07:59:02.829562 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.829532 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d0f1d8-5ff1-4469-9539-51eb8d91fbaf" containerName="test-curl-7361-f96a04" Nov 02 07:59:02.833019 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.832992 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-901598" Nov 02 07:59:02.834524 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.834501 5899 generic.go:356] "Generic (PLEG): container finished" podID="1983efdc-6a25-4e29-8a58-735284644376" containerID="d254e2cbb63f609e0ae067ccf452978db39bd9283653ee4426266ea4fe13a534" exitCode=137 Nov 02 07:59:02.834615 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.834554 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-149f84" Nov 02 07:59:02.834615 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.834594 5899 scope.go:122] "RemoveContainer" containerID="d254e2cbb63f609e0ae067ccf452978db39bd9283653ee4426266ea4fe13a534" Nov 02 07:59:02.850837 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.848984 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-901598"] Nov 02 07:59:02.853209 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.853150 5899 scope.go:122] "RemoveContainer" containerID="d254e2cbb63f609e0ae067ccf452978db39bd9283653ee4426266ea4fe13a534" Nov 02 07:59:02.853527 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:02.853506 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d254e2cbb63f609e0ae067ccf452978db39bd9283653ee4426266ea4fe13a534\": container with ID starting with d254e2cbb63f609e0ae067ccf452978db39bd9283653ee4426266ea4fe13a534 not found: ID does not exist" containerID="d254e2cbb63f609e0ae067ccf452978db39bd9283653ee4426266ea4fe13a534" Nov 02 07:59:02.853605 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.853541 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d254e2cbb63f609e0ae067ccf452978db39bd9283653ee4426266ea4fe13a534"} err="failed to get container status \"d254e2cbb63f609e0ae067ccf452978db39bd9283653ee4426266ea4fe13a534\": rpc error: code = NotFound desc = could not find container \"d254e2cbb63f609e0ae067ccf452978db39bd9283653ee4426266ea4fe13a534\": container with ID starting with d254e2cbb63f609e0ae067ccf452978db39bd9283653ee4426266ea4fe13a534 not found: ID does not exist" Nov 02 07:59:02.885961 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:02.885920 5899 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-149f84\" is forbidden: User \"system:node:ip-10-0-135-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-6.ec2.internal' and this object" podUID="1983efdc-6a25-4e29-8a58-735284644376" pod="opendatahub/test-curl-7361-149f84" Nov 02 07:59:02.899028 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:02.898944 5899 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-149f84\" is forbidden: User \"system:node:ip-10-0-135-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-6.ec2.internal' and this object" podUID="1983efdc-6a25-4e29-8a58-735284644376" pod="opendatahub/test-curl-7361-149f84" Nov 02 07:59:02.928677 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:02.928642 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkw9c\" (UniqueName: \"kubernetes.io/projected/0d432323-dc76-496c-b0b1-5120c4ff7599-kube-api-access-dkw9c\") pod \"test-curl-7361-901598\" (UID: \"0d432323-dc76-496c-b0b1-5120c4ff7599\") " pod="opendatahub/test-curl-7361-901598" Nov 02 07:59:03.029871 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:03.029839 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkw9c\" (UniqueName: \"kubernetes.io/projected/0d432323-dc76-496c-b0b1-5120c4ff7599-kube-api-access-dkw9c\") pod \"test-curl-7361-901598\" (UID: \"0d432323-dc76-496c-b0b1-5120c4ff7599\") " pod="opendatahub/test-curl-7361-901598" Nov 02 07:59:03.052948 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:03.052921 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkw9c\" (UniqueName: \"kubernetes.io/projected/0d432323-dc76-496c-b0b1-5120c4ff7599-kube-api-access-dkw9c\") pod \"test-curl-7361-901598\" (UID: \"0d432323-dc76-496c-b0b1-5120c4ff7599\") " pod="opendatahub/test-curl-7361-901598" Nov 02 07:59:03.151724 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:03.151645 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-901598" Nov 02 07:59:03.451838 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:03.451810 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-901598"] Nov 02 07:59:03.503844 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:03.503816 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:03.826298 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:03.824297 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1983efdc-6a25-4e29-8a58-735284644376" path="/var/lib/kubelet/pods/1983efdc-6a25-4e29-8a58-735284644376/volumes" Nov 02 07:59:03.843819 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:03.843785 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-901598" event={"ID":"0d432323-dc76-496c-b0b1-5120c4ff7599","Type":"ContainerStarted","Data":"7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313"} Nov 02 07:59:03.843916 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:03.843834 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-901598" event={"ID":"0d432323-dc76-496c-b0b1-5120c4ff7599","Type":"ContainerStarted","Data":"36e60bee99ef7b08da2f10a9fc0940ba8eaaa997dfb65314b242d48df3655df8"} Nov 02 07:59:03.843916 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:03.843904 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7361-f96a04" podUID="66d0f1d8-5ff1-4469-9539-51eb8d91fbaf" containerName="test-curl-7361-f96a04" containerID="cri-o://b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09" gracePeriod=2 Nov 02 07:59:03.881242 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:03.881182 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7361-901598" podStartSLOduration=1.8811479389999999 podStartE2EDuration="1.881147939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:03.880466637 +0000 UTC m=+2044.178747463" watchObservedRunningTime="2026-11-02 07:59:03.881147939 +0000 UTC m=+2044.179428765" Nov 02 07:59:04.003151 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.003124 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-22fe8c" Nov 02 07:59:04.045825 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.045732 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97370d71-3b32-4d01-8130-3bff59c4d1b0-kube-api-access-jb29q\" (UniqueName: \"kubernetes.io/projected/97370d71-3b32-4d01-8130-3bff59c4d1b0-kube-api-access-jb29q\") pod \"97370d71-3b32-4d01-8130-3bff59c4d1b0\" (UID: \"97370d71-3b32-4d01-8130-3bff59c4d1b0\") " Nov 02 07:59:04.053598 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.047810 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97370d71-3b32-4d01-8130-3bff59c4d1b0-kube-api-access-jb29q" pod "97370d71-3b32-4d01-8130-3bff59c4d1b0" (UID: "97370d71-3b32-4d01-8130-3bff59c4d1b0"). InnerVolumeSpecName "kube-api-access-jb29q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:59:04.100009 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.099918 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:04.147838 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.147810 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb29q\" (UniqueName: \"kubernetes.io/projected/97370d71-3b32-4d01-8130-3bff59c4d1b0-kube-api-access-jb29q\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:04.713834 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.713804 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7361-901598"] Nov 02 07:59:04.717960 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.717928 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7361-901598"] Nov 02 07:59:04.763022 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.762985 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:04.851965 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.851925 5899 generic.go:356] "Generic (PLEG): container finished" podID="97370d71-3b32-4d01-8130-3bff59c4d1b0" containerID="1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489" exitCode=137 Nov 02 07:59:04.851965 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.851964 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-22fe8c" Nov 02 07:59:04.852217 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.852003 5899 scope.go:122] "RemoveContainer" containerID="1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489" Nov 02 07:59:04.859298 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.859270 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7361-7d48e3"] Nov 02 07:59:04.860322 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.860098 5899 cpu_manager.go:414] "containerMap: removing container" podUID="0d432323-dc76-496c-b0b1-5120c4ff7599" containerName="test-curl-7361-901598" Nov 02 07:59:04.860322 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.860117 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d432323-dc76-496c-b0b1-5120c4ff7599" containerName="test-curl-7361-901598" Nov 02 07:59:04.860452 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.860353 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d432323-dc76-496c-b0b1-5120c4ff7599" containerName="test-curl-7361-901598" Nov 02 07:59:04.863493 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.863478 5899 scope.go:122] "RemoveContainer" containerID="1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489" Nov 02 07:59:04.863756 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:04.863736 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489\": container with ID starting with 1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489 not found: ID does not exist" containerID="1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489" Nov 02 07:59:04.863817 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.863764 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489"} err="failed to get container status \"1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489\": rpc error: code = NotFound desc = could not find container \"1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489\": container with ID starting with 1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489 not found: ID does not exist" Nov 02 07:59:04.865251 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.865236 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-7d48e3" Nov 02 07:59:04.873860 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:04.873838 5899 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-22fe8c\" is forbidden: User \"system:node:ip-10-0-135-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-6.ec2.internal' and this object" podUID="97370d71-3b32-4d01-8130-3bff59c4d1b0" pod="opendatahub/test-curl-7361-22fe8c" Nov 02 07:59:04.879865 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.879840 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-7d48e3"] Nov 02 07:59:04.922824 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:04.922787 5899 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-22fe8c\" is forbidden: User \"system:node:ip-10-0-135-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-6.ec2.internal' and this object" podUID="97370d71-3b32-4d01-8130-3bff59c4d1b0" pod="opendatahub/test-curl-7361-22fe8c" Nov 02 07:59:04.957087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:04.957056 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fch8\" (UniqueName: \"kubernetes.io/projected/4a7449f8-6e4b-44ae-aa16-e3e700cd15b0-kube-api-access-6fch8\") pod \"test-curl-7361-7d48e3\" (UID: \"4a7449f8-6e4b-44ae-aa16-e3e700cd15b0\") " pod="opendatahub/test-curl-7361-7d48e3" Nov 02 07:59:05.058833 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:05.058801 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fch8\" (UniqueName: \"kubernetes.io/projected/4a7449f8-6e4b-44ae-aa16-e3e700cd15b0-kube-api-access-6fch8\") pod \"test-curl-7361-7d48e3\" (UID: \"4a7449f8-6e4b-44ae-aa16-e3e700cd15b0\") " pod="opendatahub/test-curl-7361-7d48e3" Nov 02 07:59:05.081871 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:05.081845 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fch8\" (UniqueName: \"kubernetes.io/projected/4a7449f8-6e4b-44ae-aa16-e3e700cd15b0-kube-api-access-6fch8\") pod \"test-curl-7361-7d48e3\" (UID: \"4a7449f8-6e4b-44ae-aa16-e3e700cd15b0\") " pod="opendatahub/test-curl-7361-7d48e3" Nov 02 07:59:05.177645 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:05.177616 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-7d48e3" Nov 02 07:59:05.293438 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:05.293406 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:05.479798 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:05.479773 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-7d48e3"] Nov 02 07:59:05.819979 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:05.819897 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97370d71-3b32-4d01-8130-3bff59c4d1b0" path="/var/lib/kubelet/pods/97370d71-3b32-4d01-8130-3bff59c4d1b0/volumes" Nov 02 07:59:05.859725 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:05.859689 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-7d48e3" event={"ID":"4a7449f8-6e4b-44ae-aa16-e3e700cd15b0","Type":"ContainerStarted","Data":"9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0"} Nov 02 07:59:05.859881 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:05.859738 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-7d48e3" event={"ID":"4a7449f8-6e4b-44ae-aa16-e3e700cd15b0","Type":"ContainerStarted","Data":"382b66f4ff112e09e8220680f261ea6edcd0171ffbfc733c58b1fd0be3aa8347"} Nov 02 07:59:05.860889 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:05.860863 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7361-901598" podUID="0d432323-dc76-496c-b0b1-5120c4ff7599" containerName="test-curl-7361-901598" containerID="cri-o://7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313" gracePeriod=2 Nov 02 07:59:05.874942 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:59:05.874916 5899 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66d0f1d8_5ff1_4469_9539_51eb8d91fbaf.slice/crio-375106be7ab934208a6524930b2348299001738354740a0a09b452af96ab28fd/cpuset.cpus.effective": open /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66d0f1d8_5ff1_4469_9539_51eb8d91fbaf.slice/crio-375106be7ab934208a6524930b2348299001738354740a0a09b452af96ab28fd/cpuset.cpus.effective: no such device Nov 02 07:59:05.900355 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:05.899826 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7361-7d48e3" podStartSLOduration=1.899808186 podStartE2EDuration="1.899808186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:05.896728969 +0000 UTC m=+2046.195009795" watchObservedRunningTime="2026-11-02 07:59:05.899808186 +0000 UTC m=+2046.198089013" Nov 02 07:59:06.018479 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.018453 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-f96a04" Nov 02 07:59:06.073044 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.072970 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d0f1d8-5ff1-4469-9539-51eb8d91fbaf-kube-api-access-xm5fk\" (UniqueName: \"kubernetes.io/projected/66d0f1d8-5ff1-4469-9539-51eb8d91fbaf-kube-api-access-xm5fk\") pod \"66d0f1d8-5ff1-4469-9539-51eb8d91fbaf\" (UID: \"66d0f1d8-5ff1-4469-9539-51eb8d91fbaf\") " Nov 02 07:59:06.075248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.075217 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d0f1d8-5ff1-4469-9539-51eb8d91fbaf-kube-api-access-xm5fk" pod "66d0f1d8-5ff1-4469-9539-51eb8d91fbaf" (UID: "66d0f1d8-5ff1-4469-9539-51eb8d91fbaf"). InnerVolumeSpecName "kube-api-access-xm5fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:59:06.181504 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.181467 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm5fk\" (UniqueName: \"kubernetes.io/projected/66d0f1d8-5ff1-4469-9539-51eb8d91fbaf-kube-api-access-xm5fk\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:06.730668 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.730625 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7361-7d48e3"] Nov 02 07:59:06.733510 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.733476 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7361-7d48e3"] Nov 02 07:59:06.868521 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.868484 5899 generic.go:356] "Generic (PLEG): container finished" podID="66d0f1d8-5ff1-4469-9539-51eb8d91fbaf" containerID="b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09" exitCode=137 Nov 02 07:59:06.868710 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.868528 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-f96a04" Nov 02 07:59:06.868710 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.868580 5899 scope.go:122] "RemoveContainer" containerID="b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09" Nov 02 07:59:06.873100 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.873076 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7361-1277d5"] Nov 02 07:59:06.873794 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.873771 5899 cpu_manager.go:414] "containerMap: removing container" podUID="4a7449f8-6e4b-44ae-aa16-e3e700cd15b0" containerName="test-curl-7361-7d48e3" Nov 02 07:59:06.873794 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.873792 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7449f8-6e4b-44ae-aa16-e3e700cd15b0" containerName="test-curl-7361-7d48e3" Nov 02 07:59:06.874006 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.873993 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7449f8-6e4b-44ae-aa16-e3e700cd15b0" containerName="test-curl-7361-7d48e3" Nov 02 07:59:06.878433 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.878415 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-1277d5" Nov 02 07:59:06.879288 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.879272 5899 scope.go:122] "RemoveContainer" containerID="b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09" Nov 02 07:59:06.879567 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:06.879551 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09\": container with ID starting with b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09 not found: ID does not exist" containerID="b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09" Nov 02 07:59:06.879638 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.879574 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09"} err="failed to get container status \"b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09\": rpc error: code = NotFound desc = could not find container \"b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09\": container with ID starting with b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09 not found: ID does not exist" Nov 02 07:59:06.891390 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:06.891360 5899 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-f96a04\" is forbidden: User \"system:node:ip-10-0-135-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-6.ec2.internal' and this object" podUID="66d0f1d8-5ff1-4469-9539-51eb8d91fbaf" pod="opendatahub/test-curl-7361-f96a04" Nov 02 07:59:06.897625 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.897597 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-1277d5"] Nov 02 07:59:06.904521 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:06.904477 5899 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-f96a04\" is forbidden: User \"system:node:ip-10-0-135-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-6.ec2.internal' and this object" podUID="66d0f1d8-5ff1-4469-9539-51eb8d91fbaf" pod="opendatahub/test-curl-7361-f96a04" Nov 02 07:59:06.992854 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:06.992767 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvsnq\" (UniqueName: \"kubernetes.io/projected/f8f68682-32a1-4dbc-9098-cf8c48cff578-kube-api-access-jvsnq\") pod \"test-curl-7361-1277d5\" (UID: \"f8f68682-32a1-4dbc-9098-cf8c48cff578\") " pod="opendatahub/test-curl-7361-1277d5" Nov 02 07:59:07.094327 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:07.094283 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsnq\" (UniqueName: \"kubernetes.io/projected/f8f68682-32a1-4dbc-9098-cf8c48cff578-kube-api-access-jvsnq\") pod \"test-curl-7361-1277d5\" (UID: \"f8f68682-32a1-4dbc-9098-cf8c48cff578\") " pod="opendatahub/test-curl-7361-1277d5" Nov 02 07:59:07.117375 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:07.117344 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvsnq\" (UniqueName: \"kubernetes.io/projected/f8f68682-32a1-4dbc-9098-cf8c48cff578-kube-api-access-jvsnq\") pod \"test-curl-7361-1277d5\" (UID: \"f8f68682-32a1-4dbc-9098-cf8c48cff578\") " pod="opendatahub/test-curl-7361-1277d5" Nov 02 07:59:07.192686 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:07.192655 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-1277d5" Nov 02 07:59:07.218010 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:07.217983 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:07.498526 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:07.498496 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7361-1277d5"] Nov 02 07:59:07.822036 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:07.821999 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d0f1d8-5ff1-4469-9539-51eb8d91fbaf" path="/var/lib/kubelet/pods/66d0f1d8-5ff1-4469-9539-51eb8d91fbaf/volumes" Nov 02 07:59:07.881117 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:07.881087 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-1277d5" event={"ID":"f8f68682-32a1-4dbc-9098-cf8c48cff578","Type":"ContainerStarted","Data":"66bb15ad18b9de042f7178fbbba8e80ca06bfc6f8b29483b6091951512c5d8cd"} Nov 02 07:59:07.881263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:07.881131 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7361-1277d5" event={"ID":"f8f68682-32a1-4dbc-9098-cf8c48cff578","Type":"ContainerStarted","Data":"de7fed29413ab71582472eb3fc65a793c3f0ce6ff9d85499186c50bc381ed6cd"} Nov 02 07:59:07.882898 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:07.882877 5899 generic.go:356] "Generic (PLEG): container finished" podID="c1f03252-734a-44ad-9c2a-4072303b9a09" containerID="5f036726866260a4f0aa7f2c289ba6b46b09b5b5f37ae37f576aa786948e35d1" exitCode=0 Nov 02 07:59:07.883015 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:07.882940 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" event={"ID":"c1f03252-734a-44ad-9c2a-4072303b9a09","Type":"ContainerDied","Data":"5f036726866260a4f0aa7f2c289ba6b46b09b5b5f37ae37f576aa786948e35d1"} Nov 02 07:59:07.883061 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:07.883026 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7361-7d48e3" podUID="4a7449f8-6e4b-44ae-aa16-e3e700cd15b0" containerName="test-curl-7361-7d48e3" containerID="cri-o://9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0" gracePeriod=2 Nov 02 07:59:07.920919 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:07.920867 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7361-1277d5" podStartSLOduration=1.9208499620000001 podStartE2EDuration="1.920849962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:07.918596653 +0000 UTC m=+2048.216877478" watchObservedRunningTime="2026-11-02 07:59:07.920849962 +0000 UTC m=+2048.219130789" Nov 02 07:59:07.934050 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:07.934020 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:59:07.960813 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:07.960756 5899 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97370d71_3b32_4d01_8130_3bff59c4d1b0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97370d71_3b32_4d01_8130_3bff59c4d1b0.slice/crio-1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66d0f1d8_5ff1_4469_9539_51eb8d91fbaf.slice/crio-b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97370d71_3b32_4d01_8130_3bff59c4d1b0.slice/crio-9fe9d828b542cf20371bae94b607296ec19c0c7f88ccf17efa5e83bc87300db1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66d0f1d8_5ff1_4469_9539_51eb8d91fbaf.slice/crio-conmon-b5de94e7fb21dfb5e98aa205253d29551ad2cf63010b4b42f47adb8a604a3a09.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97370d71_3b32_4d01_8130_3bff59c4d1b0.slice/crio-conmon-1636d2c5f45aac1556c622a2996ef0e458ddce4bf26f465d66d6037457c5d489.scope\": RecentStats: unable to find data in memory cache]" Nov 02 07:59:08.019669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.018722 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-socket\") pod \"c1f03252-734a-44ad-9c2a-4072303b9a09\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " Nov 02 07:59:08.019669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.018779 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-token\" (UniqueName: \"kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-token\") pod \"c1f03252-734a-44ad-9c2a-4072303b9a09\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " Nov 02 07:59:08.019669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.018857 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f03252-734a-44ad-9c2a-4072303b9a09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f03252-734a-44ad-9c2a-4072303b9a09-istiod-ca-cert\") pod \"c1f03252-734a-44ad-9c2a-4072303b9a09\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " Nov 02 07:59:08.019669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.018891 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-podinfo\") pod \"c1f03252-734a-44ad-9c2a-4072303b9a09\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " Nov 02 07:59:08.019669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.018947 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-kube-api-access-929jf\" (UniqueName: \"kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-kube-api-access-929jf\") pod \"c1f03252-734a-44ad-9c2a-4072303b9a09\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " Nov 02 07:59:08.019669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.019094 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-credential-socket\") pod \"c1f03252-734a-44ad-9c2a-4072303b9a09\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " Nov 02 07:59:08.019669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.019269 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-certs\") pod \"c1f03252-734a-44ad-9c2a-4072303b9a09\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " Nov 02 07:59:08.019669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.019324 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-envoy\") pod \"c1f03252-734a-44ad-9c2a-4072303b9a09\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " Nov 02 07:59:08.019669 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.019375 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-data\") pod \"c1f03252-734a-44ad-9c2a-4072303b9a09\" (UID: \"c1f03252-734a-44ad-9c2a-4072303b9a09\") " Nov 02 07:59:08.020295 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.020261 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-certs" pod "c1f03252-734a-44ad-9c2a-4072303b9a09" (UID: "c1f03252-734a-44ad-9c2a-4072303b9a09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:59:08.020569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.020508 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-credential-socket" pod "c1f03252-734a-44ad-9c2a-4072303b9a09" (UID: "c1f03252-734a-44ad-9c2a-4072303b9a09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:59:08.020731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.020694 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-data" pod "c1f03252-734a-44ad-9c2a-4072303b9a09" (UID: "c1f03252-734a-44ad-9c2a-4072303b9a09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:59:08.020900 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.020873 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-socket" pod "c1f03252-734a-44ad-9c2a-4072303b9a09" (UID: "c1f03252-734a-44ad-9c2a-4072303b9a09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:59:08.021256 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.021223 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f03252-734a-44ad-9c2a-4072303b9a09-istiod-ca-cert" pod "c1f03252-734a-44ad-9c2a-4072303b9a09" (UID: "c1f03252-734a-44ad-9c2a-4072303b9a09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:59:08.023487 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.023461 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-token" pod "c1f03252-734a-44ad-9c2a-4072303b9a09" (UID: "c1f03252-734a-44ad-9c2a-4072303b9a09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:59:08.023569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.023512 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-podinfo" pod "c1f03252-734a-44ad-9c2a-4072303b9a09" (UID: "c1f03252-734a-44ad-9c2a-4072303b9a09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:59:08.023569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.023554 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-envoy" pod "c1f03252-734a-44ad-9c2a-4072303b9a09" (UID: "c1f03252-734a-44ad-9c2a-4072303b9a09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:59:08.024359 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.024338 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-kube-api-access-929jf" pod "c1f03252-734a-44ad-9c2a-4072303b9a09" (UID: "c1f03252-734a-44ad-9c2a-4072303b9a09"). InnerVolumeSpecName "kube-api-access-929jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:59:08.076956 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.076928 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-901598" Nov 02 07:59:08.121715 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.121685 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:08.121715 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.121712 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:08.121715 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.121721 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:08.122186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.121731 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:08.122186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.121739 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f03252-734a-44ad-9c2a-4072303b9a09-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:08.122186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.121746 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f03252-734a-44ad-9c2a-4072303b9a09-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:08.122186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.121756 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-929jf\" (UniqueName: \"kubernetes.io/projected/c1f03252-734a-44ad-9c2a-4072303b9a09-kube-api-access-929jf\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:08.122186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.121766 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:08.122186 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.121774 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f03252-734a-44ad-9c2a-4072303b9a09-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:08.223196 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.223136 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d432323-dc76-496c-b0b1-5120c4ff7599-kube-api-access-dkw9c\" (UniqueName: \"kubernetes.io/projected/0d432323-dc76-496c-b0b1-5120c4ff7599-kube-api-access-dkw9c\") pod \"0d432323-dc76-496c-b0b1-5120c4ff7599\" (UID: \"0d432323-dc76-496c-b0b1-5120c4ff7599\") " Nov 02 07:59:08.225005 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.224981 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d432323-dc76-496c-b0b1-5120c4ff7599-kube-api-access-dkw9c" pod "0d432323-dc76-496c-b0b1-5120c4ff7599" (UID: "0d432323-dc76-496c-b0b1-5120c4ff7599"). InnerVolumeSpecName "kube-api-access-dkw9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:59:08.325053 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.324961 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkw9c\" (UniqueName: \"kubernetes.io/projected/0d432323-dc76-496c-b0b1-5120c4ff7599-kube-api-access-dkw9c\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:08.760707 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.760672 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7361-1277d5"] Nov 02 07:59:08.767737 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.767706 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7361-1277d5"] Nov 02 07:59:08.890788 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.890752 5899 generic.go:356] "Generic (PLEG): container finished" podID="0d432323-dc76-496c-b0b1-5120c4ff7599" containerID="7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313" exitCode=137 Nov 02 07:59:08.890972 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.890813 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-901598" Nov 02 07:59:08.890972 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.890851 5899 scope.go:122] "RemoveContainer" containerID="7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313" Nov 02 07:59:08.892511 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.892486 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" Nov 02 07:59:08.892611 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.892562 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg" event={"ID":"c1f03252-734a-44ad-9c2a-4072303b9a09","Type":"ContainerDied","Data":"307341106a3930e58c41f4f1a39a38e687194b42ac6999fe0281341b19103c04"} Nov 02 07:59:08.901937 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.901916 5899 scope.go:122] "RemoveContainer" containerID="7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313" Nov 02 07:59:08.902271 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:08.902246 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313\": container with ID starting with 7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313 not found: ID does not exist" containerID="7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313" Nov 02 07:59:08.902368 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.902283 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313"} err="failed to get container status \"7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313\": rpc error: code = NotFound desc = could not find container \"7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313\": container with ID starting with 7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313 not found: ID does not exist" Nov 02 07:59:08.902368 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.902308 5899 scope.go:122] "RemoveContainer" containerID="5f036726866260a4f0aa7f2c289ba6b46b09b5b5f37ae37f576aa786948e35d1" Nov 02 07:59:08.921657 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:08.921627 5899 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-901598\" is forbidden: User \"system:node:ip-10-0-135-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-6.ec2.internal' and this object" podUID="0d432323-dc76-496c-b0b1-5120c4ff7599" pod="opendatahub/test-curl-7361-901598" Nov 02 07:59:08.958063 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.958037 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg"] Nov 02 07:59:08.969716 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:08.969694 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-4fpwg"] Nov 02 07:59:09.005370 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:09.005344 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:09.820689 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:09.820654 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d432323-dc76-496c-b0b1-5120c4ff7599" path="/var/lib/kubelet/pods/0d432323-dc76-496c-b0b1-5120c4ff7599/volumes" Nov 02 07:59:09.821365 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:09.821345 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f03252-734a-44ad-9c2a-4072303b9a09" path="/var/lib/kubelet/pods/c1f03252-734a-44ad-9c2a-4072303b9a09/volumes" Nov 02 07:59:09.828920 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:09.828894 5899 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-901598\" is forbidden: User \"system:node:ip-10-0-135-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-6.ec2.internal' and this object" podUID="0d432323-dc76-496c-b0b1-5120c4ff7599" pod="opendatahub/test-curl-7361-901598" Nov 02 07:59:09.901958 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:09.901927 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7361-1277d5" podUID="f8f68682-32a1-4dbc-9098-cf8c48cff578" containerName="test-curl-7361-1277d5" containerID="cri-o://66bb15ad18b9de042f7178fbbba8e80ca06bfc6f8b29483b6091951512c5d8cd" gracePeriod=2 Nov 02 07:59:10.087776 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:10.087718 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-7d48e3" Nov 02 07:59:10.248926 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:10.248890 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7449f8-6e4b-44ae-aa16-e3e700cd15b0-kube-api-access-6fch8\" (UniqueName: \"kubernetes.io/projected/4a7449f8-6e4b-44ae-aa16-e3e700cd15b0-kube-api-access-6fch8\") pod \"4a7449f8-6e4b-44ae-aa16-e3e700cd15b0\" (UID: \"4a7449f8-6e4b-44ae-aa16-e3e700cd15b0\") " Nov 02 07:59:10.250969 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:10.250937 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7449f8-6e4b-44ae-aa16-e3e700cd15b0-kube-api-access-6fch8" pod "4a7449f8-6e4b-44ae-aa16-e3e700cd15b0" (UID: "4a7449f8-6e4b-44ae-aa16-e3e700cd15b0"). InnerVolumeSpecName "kube-api-access-6fch8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:59:10.271095 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:10.271073 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:10.350860 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:10.350790 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fch8\" (UniqueName: \"kubernetes.io/projected/4a7449f8-6e4b-44ae-aa16-e3e700cd15b0-kube-api-access-6fch8\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:10.857569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:10.857518 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:10.911288 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:10.911257 5899 generic.go:356] "Generic (PLEG): container finished" podID="4a7449f8-6e4b-44ae-aa16-e3e700cd15b0" containerID="9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0" exitCode=137 Nov 02 07:59:10.911453 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:10.911320 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-7d48e3" Nov 02 07:59:10.911453 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:10.911361 5899 scope.go:122] "RemoveContainer" containerID="9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0" Nov 02 07:59:10.922742 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:10.922712 5899 scope.go:122] "RemoveContainer" containerID="9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0" Nov 02 07:59:10.922980 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:10.922961 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0\": container with ID starting with 9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0 not found: ID does not exist" containerID="9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0" Nov 02 07:59:10.923051 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:10.922988 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0"} err="failed to get container status \"9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0\": rpc error: code = NotFound desc = could not find container \"9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0\": container with ID starting with 9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0 not found: ID does not exist" Nov 02 07:59:10.935419 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:10.935386 5899 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-7d48e3\" is forbidden: User \"system:node:ip-10-0-135-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-6.ec2.internal' and this object" podUID="4a7449f8-6e4b-44ae-aa16-e3e700cd15b0" pod="opendatahub/test-curl-7361-7d48e3" Nov 02 07:59:11.043627 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:11.043590 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:11.338791 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:11.338742 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:11.822457 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:11.822415 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a7449f8-6e4b-44ae-aa16-e3e700cd15b0" path="/var/lib/kubelet/pods/4a7449f8-6e4b-44ae-aa16-e3e700cd15b0/volumes" Nov 02 07:59:11.950959 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:11.950917 5899 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8f68682_32a1_4dbc_9098_cf8c48cff578.slice/crio-de7fed29413ab71582472eb3fc65a793c3f0ce6ff9d85499186c50bc381ed6cd\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a7449f8_6e4b_44ae_aa16_e3e700cd15b0.slice/crio-conmon-9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d432323_dc76_496c_b0b1_5120c4ff7599.slice/crio-conmon-7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a7449f8_6e4b_44ae_aa16_e3e700cd15b0.slice/crio-9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d432323_dc76_496c_b0b1_5120c4ff7599.slice/crio-7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313.scope\": RecentStats: unable to find data in memory cache]" Nov 02 07:59:11.951780 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:11.951731 5899 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d432323_dc76_496c_b0b1_5120c4ff7599.slice/crio-conmon-7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8f68682_32a1_4dbc_9098_cf8c48cff578.slice/crio-de7fed29413ab71582472eb3fc65a793c3f0ce6ff9d85499186c50bc381ed6cd\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a7449f8_6e4b_44ae_aa16_e3e700cd15b0.slice/crio-9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d432323_dc76_496c_b0b1_5120c4ff7599.slice/crio-7d26eb94a2e8e77d00f9a2e5030c38c7382671a7844a2e7ca218b67e1de93313.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a7449f8_6e4b_44ae_aa16_e3e700cd15b0.slice/crio-conmon-9d67516ec34e732861ccdd2eb4dbedc9476610e239c009fd243cbcc267293cb0.scope\": RecentStats: unable to find data in memory cache]" Nov 02 07:59:12.142624 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:12.142594 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-1277d5" Nov 02 07:59:12.183335 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:12.183303 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f68682-32a1-4dbc-9098-cf8c48cff578-kube-api-access-jvsnq\" (UniqueName: \"kubernetes.io/projected/f8f68682-32a1-4dbc-9098-cf8c48cff578-kube-api-access-jvsnq\") pod \"f8f68682-32a1-4dbc-9098-cf8c48cff578\" (UID: \"f8f68682-32a1-4dbc-9098-cf8c48cff578\") " Nov 02 07:59:12.185640 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:12.185610 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f68682-32a1-4dbc-9098-cf8c48cff578-kube-api-access-jvsnq" pod "f8f68682-32a1-4dbc-9098-cf8c48cff578" (UID: "f8f68682-32a1-4dbc-9098-cf8c48cff578"). InnerVolumeSpecName "kube-api-access-jvsnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:59:12.284446 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:12.284409 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvsnq\" (UniqueName: \"kubernetes.io/projected/f8f68682-32a1-4dbc-9098-cf8c48cff578-kube-api-access-jvsnq\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:12.721172 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:12.721139 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:12.927689 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:12.927656 5899 generic.go:356] "Generic (PLEG): container finished" podID="f8f68682-32a1-4dbc-9098-cf8c48cff578" containerID="66bb15ad18b9de042f7178fbbba8e80ca06bfc6f8b29483b6091951512c5d8cd" exitCode=137 Nov 02 07:59:12.927829 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:12.927701 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7361-1277d5" Nov 02 07:59:12.927829 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:12.927751 5899 scope.go:122] "RemoveContainer" containerID="66bb15ad18b9de042f7178fbbba8e80ca06bfc6f8b29483b6091951512c5d8cd" Nov 02 07:59:12.937471 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:12.937454 5899 scope.go:122] "RemoveContainer" containerID="66bb15ad18b9de042f7178fbbba8e80ca06bfc6f8b29483b6091951512c5d8cd" Nov 02 07:59:12.937764 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:12.937741 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66bb15ad18b9de042f7178fbbba8e80ca06bfc6f8b29483b6091951512c5d8cd\": container with ID starting with 66bb15ad18b9de042f7178fbbba8e80ca06bfc6f8b29483b6091951512c5d8cd not found: ID does not exist" containerID="66bb15ad18b9de042f7178fbbba8e80ca06bfc6f8b29483b6091951512c5d8cd" Nov 02 07:59:12.937865 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:12.937770 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66bb15ad18b9de042f7178fbbba8e80ca06bfc6f8b29483b6091951512c5d8cd"} err="failed to get container status \"66bb15ad18b9de042f7178fbbba8e80ca06bfc6f8b29483b6091951512c5d8cd\": rpc error: code = NotFound desc = could not find container \"66bb15ad18b9de042f7178fbbba8e80ca06bfc6f8b29483b6091951512c5d8cd\": container with ID starting with 66bb15ad18b9de042f7178fbbba8e80ca06bfc6f8b29483b6091951512c5d8cd not found: ID does not exist" Nov 02 07:59:12.949472 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:12.949443 5899 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7361-1277d5\" is forbidden: User \"system:node:ip-10-0-135-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-6.ec2.internal' and this object" podUID="f8f68682-32a1-4dbc-9098-cf8c48cff578" pod="opendatahub/test-curl-7361-1277d5" Nov 02 07:59:13.142277 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:13.142243 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:59:13.820269 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:13.820238 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f68682-32a1-4dbc-9098-cf8c48cff578" path="/var/lib/kubelet/pods/f8f68682-32a1-4dbc-9098-cf8c48cff578/volumes" Nov 02 07:59:14.267642 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:14.267610 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:14.432832 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:14.432804 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:14.678768 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:14.678684 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:16.036248 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:16.036208 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:16.179482 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:16.179452 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:16.795810 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:16.795781 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:18.632794 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:18.632757 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:18.881063 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:18.881031 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j"] Nov 02 07:59:18.881280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:18.881259 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" podUID="122a1a15-edb5-43b0-a9fe-9d95381ee889" containerName="istio-proxy" containerID="cri-o://4160f017a3fe59abec8cf7d293b676dd701652149bc95a968dab736070245a71" gracePeriod=30 Nov 02 07:59:19.079822 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:19.079783 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:20.202923 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:20.202865 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:20.609616 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:20.609548 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:21.503647 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:21.503606 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:21.634909 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:21.634875 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:21.774389 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:21.774316 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:22.282532 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:22.282491 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:22.439718 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:22.439682 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:22.581296 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:22.581218 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:22.988653 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:22.988621 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:24.030152 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.030117 5899 generic.go:356] "Generic (PLEG): container finished" podID="122a1a15-edb5-43b0-a9fe-9d95381ee889" containerID="4160f017a3fe59abec8cf7d293b676dd701652149bc95a968dab736070245a71" exitCode=0 Nov 02 07:59:24.030508 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.030199 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" event={"ID":"122a1a15-edb5-43b0-a9fe-9d95381ee889","Type":"ContainerDied","Data":"4160f017a3fe59abec8cf7d293b676dd701652149bc95a968dab736070245a71"} Nov 02 07:59:24.153293 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.153234 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:59:24.245983 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.245945 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-envoy\") pod \"122a1a15-edb5-43b0-a9fe-9d95381ee889\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " Nov 02 07:59:24.246183 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.246051 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-data\") pod \"122a1a15-edb5-43b0-a9fe-9d95381ee889\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " Nov 02 07:59:24.246252 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.246195 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122a1a15-edb5-43b0-a9fe-9d95381ee889-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122a1a15-edb5-43b0-a9fe-9d95381ee889-istiod-ca-cert\") pod \"122a1a15-edb5-43b0-a9fe-9d95381ee889\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " Nov 02 07:59:24.246252 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.246240 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-podinfo\") pod \"122a1a15-edb5-43b0-a9fe-9d95381ee889\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " Nov 02 07:59:24.246426 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.246399 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-data" pod "122a1a15-edb5-43b0-a9fe-9d95381ee889" (UID: "122a1a15-edb5-43b0-a9fe-9d95381ee889"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:59:24.246709 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.246683 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122a1a15-edb5-43b0-a9fe-9d95381ee889-istiod-ca-cert" pod "122a1a15-edb5-43b0-a9fe-9d95381ee889" (UID: "122a1a15-edb5-43b0-a9fe-9d95381ee889"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:59:24.247280 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.247257 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-socket\") pod \"122a1a15-edb5-43b0-a9fe-9d95381ee889\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " Nov 02 07:59:24.247347 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.247332 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-kube-api-access-kqvx6\" (UniqueName: \"kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-kube-api-access-kqvx6\") pod \"122a1a15-edb5-43b0-a9fe-9d95381ee889\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " Nov 02 07:59:24.247437 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.247416 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-certs\") pod \"122a1a15-edb5-43b0-a9fe-9d95381ee889\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " Nov 02 07:59:24.247689 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.247472 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-credential-socket\") pod \"122a1a15-edb5-43b0-a9fe-9d95381ee889\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " Nov 02 07:59:24.247689 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.247509 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-token\" (UniqueName: \"kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-token\") pod \"122a1a15-edb5-43b0-a9fe-9d95381ee889\" (UID: \"122a1a15-edb5-43b0-a9fe-9d95381ee889\") " Nov 02 07:59:24.247689 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.247582 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-socket" pod "122a1a15-edb5-43b0-a9fe-9d95381ee889" (UID: "122a1a15-edb5-43b0-a9fe-9d95381ee889"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:59:24.247874 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.247809 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-certs" pod "122a1a15-edb5-43b0-a9fe-9d95381ee889" (UID: "122a1a15-edb5-43b0-a9fe-9d95381ee889"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:59:24.248148 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.248125 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/122a1a15-edb5-43b0-a9fe-9d95381ee889-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:24.248253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.248155 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:24.248253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.248185 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:24.248253 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.248197 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:24.248566 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.248538 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-credential-socket" pod "122a1a15-edb5-43b0-a9fe-9d95381ee889" (UID: "122a1a15-edb5-43b0-a9fe-9d95381ee889"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:59:24.249382 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.249354 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-podinfo" pod "122a1a15-edb5-43b0-a9fe-9d95381ee889" (UID: "122a1a15-edb5-43b0-a9fe-9d95381ee889"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:59:24.249982 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.249950 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-kube-api-access-kqvx6" pod "122a1a15-edb5-43b0-a9fe-9d95381ee889" (UID: "122a1a15-edb5-43b0-a9fe-9d95381ee889"). InnerVolumeSpecName "kube-api-access-kqvx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:59:24.250120 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.250096 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-token" pod "122a1a15-edb5-43b0-a9fe-9d95381ee889" (UID: "122a1a15-edb5-43b0-a9fe-9d95381ee889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:59:24.251336 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.251313 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-envoy" pod "122a1a15-edb5-43b0-a9fe-9d95381ee889" (UID: "122a1a15-edb5-43b0-a9fe-9d95381ee889"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:59:24.350038 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.350002 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:24.350259 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.350044 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqvx6\" (UniqueName: \"kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-kube-api-access-kqvx6\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:24.350259 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.350060 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:24.350259 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.350074 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:24.350259 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.350089 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/122a1a15-edb5-43b0-a9fe-9d95381ee889-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:24.867024 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:24.866988 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:25.037455 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:25.037424 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" Nov 02 07:59:25.037903 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:25.037457 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j" event={"ID":"122a1a15-edb5-43b0-a9fe-9d95381ee889","Type":"ContainerDied","Data":"ebfd941475a33285a8b239ae9298ca48066adf20c9704883d572236e88794153"} Nov 02 07:59:25.037903 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:25.037544 5899 scope.go:122] "RemoveContainer" containerID="4160f017a3fe59abec8cf7d293b676dd701652149bc95a968dab736070245a71" Nov 02 07:59:25.091972 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:25.091943 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j"] Nov 02 07:59:25.103658 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:25.103629 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-chs7j"] Nov 02 07:59:25.819123 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:25.819086 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122a1a15-edb5-43b0-a9fe-9d95381ee889" path="/var/lib/kubelet/pods/122a1a15-edb5-43b0-a9fe-9d95381ee889/volumes" Nov 02 07:59:26.484087 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:26.484042 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:26.685385 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:26.685353 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:26.864124 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:26.864023 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:28.202081 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:28.202046 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:28.361650 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:28.361598 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:28.709035 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:28.709000 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:29.303899 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:29.303865 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:30.329289 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.329259 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb"] Nov 02 07:59:30.329688 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.329494 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" podUID="0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3" containerName="maas-api" containerID="cri-o://b4b1df4e3f9681c933660e41340a240f9836e9d7f9073844a5fcf97593b6e7f3" gracePeriod=30 Nov 02 07:59:30.457936 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.457905 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:30.643170 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.643137 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:59:30.733173 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.733122 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-kube-api-access-bj5wv\" (UniqueName: \"kubernetes.io/projected/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-kube-api-access-bj5wv\") pod \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " Nov 02 07:59:30.733378 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.733220 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-metrics-certs\") pod \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " Nov 02 07:59:30.733378 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.733330 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-openshift-service-ca\") pod \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " Nov 02 07:59:30.733499 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.733419 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-maas-api-tls\") pod \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\" (UID: \"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3\") " Nov 02 07:59:30.733976 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.733948 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-openshift-service-ca" pod "0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3" (UID: "0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:59:30.736255 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.736220 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-metrics-certs" pod "0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3" (UID: "0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:59:30.736362 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.736271 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-kube-api-access-bj5wv" pod "0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3" (UID: "0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3"). InnerVolumeSpecName "kube-api-access-bj5wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:59:30.736583 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.736545 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-maas-api-tls" pod "0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3" (UID: "0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:59:30.835780 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.835745 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj5wv\" (UniqueName: \"kubernetes.io/projected/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-kube-api-access-bj5wv\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:30.835962 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.835787 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:30.835962 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.835811 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:30.835962 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:30.835832 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:31.077378 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:31.077346 5899 generic.go:356] "Generic (PLEG): container finished" podID="0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3" containerID="b4b1df4e3f9681c933660e41340a240f9836e9d7f9073844a5fcf97593b6e7f3" exitCode=0 Nov 02 07:59:31.077554 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:31.077405 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" Nov 02 07:59:31.077554 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:31.077426 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" event={"ID":"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3","Type":"ContainerDied","Data":"b4b1df4e3f9681c933660e41340a240f9836e9d7f9073844a5fcf97593b6e7f3"} Nov 02 07:59:31.077554 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:31.077512 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb" event={"ID":"0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3","Type":"ContainerDied","Data":"8ab67ae63213356a4b6a01b91a1e5273fdeecb9c7223a899aaf92728bed63006"} Nov 02 07:59:31.077554 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:31.077541 5899 scope.go:122] "RemoveContainer" containerID="b4b1df4e3f9681c933660e41340a240f9836e9d7f9073844a5fcf97593b6e7f3" Nov 02 07:59:31.089237 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:31.089221 5899 scope.go:122] "RemoveContainer" containerID="b4b1df4e3f9681c933660e41340a240f9836e9d7f9073844a5fcf97593b6e7f3" Nov 02 07:59:31.089497 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:31.089475 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b1df4e3f9681c933660e41340a240f9836e9d7f9073844a5fcf97593b6e7f3\": container with ID starting with b4b1df4e3f9681c933660e41340a240f9836e9d7f9073844a5fcf97593b6e7f3 not found: ID does not exist" containerID="b4b1df4e3f9681c933660e41340a240f9836e9d7f9073844a5fcf97593b6e7f3" Nov 02 07:59:31.089591 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:31.089503 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b1df4e3f9681c933660e41340a240f9836e9d7f9073844a5fcf97593b6e7f3"} err="failed to get container status \"b4b1df4e3f9681c933660e41340a240f9836e9d7f9073844a5fcf97593b6e7f3\": rpc error: code = NotFound desc = could not find container \"b4b1df4e3f9681c933660e41340a240f9836e9d7f9073844a5fcf97593b6e7f3\": container with ID starting with b4b1df4e3f9681c933660e41340a240f9836e9d7f9073844a5fcf97593b6e7f3 not found: ID does not exist" Nov 02 07:59:31.132683 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:31.132655 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb"] Nov 02 07:59:31.144350 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:31.144221 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ac716b-6fc9547788-4cbxb"] Nov 02 07:59:31.154902 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:31.154881 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:31.821805 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:31.821760 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3" path="/var/lib/kubelet/pods/0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3/volumes" Nov 02 07:59:33.254882 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:33.254838 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:34.275126 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.275094 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7352-76e02c"] Nov 02 07:59:34.275622 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.275608 5899 cpu_manager.go:414] "containerMap: removing container" podUID="c1f03252-734a-44ad-9c2a-4072303b9a09" containerName="istio-proxy" Nov 02 07:59:34.275670 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.275623 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f03252-734a-44ad-9c2a-4072303b9a09" containerName="istio-proxy" Nov 02 07:59:34.275670 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.275662 5899 cpu_manager.go:414] "containerMap: removing container" podUID="f8f68682-32a1-4dbc-9098-cf8c48cff578" containerName="test-curl-7361-1277d5" Nov 02 07:59:34.275731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.275670 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f68682-32a1-4dbc-9098-cf8c48cff578" containerName="test-curl-7361-1277d5" Nov 02 07:59:34.275731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.275688 5899 cpu_manager.go:414] "containerMap: removing container" podUID="122a1a15-edb5-43b0-a9fe-9d95381ee889" containerName="istio-proxy" Nov 02 07:59:34.275731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.275693 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122a1a15-edb5-43b0-a9fe-9d95381ee889" containerName="istio-proxy" Nov 02 07:59:34.275731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.275708 5899 cpu_manager.go:414] "containerMap: removing container" podUID="0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3" containerName="maas-api" Nov 02 07:59:34.275731 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.275712 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3" containerName="maas-api" Nov 02 07:59:34.275873 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.275802 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f68682-32a1-4dbc-9098-cf8c48cff578" containerName="test-curl-7361-1277d5" Nov 02 07:59:34.275873 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.275812 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="122a1a15-edb5-43b0-a9fe-9d95381ee889" containerName="istio-proxy" Nov 02 07:59:34.275873 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.275821 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="0406f5ad-ef2c-4cb0-b8dd-cb3d98d717b3" containerName="maas-api" Nov 02 07:59:34.275873 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.275828 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f03252-734a-44ad-9c2a-4072303b9a09" containerName="istio-proxy" Nov 02 07:59:34.280964 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.280942 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-76e02c" Nov 02 07:59:34.299277 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.299244 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-76e02c"] Nov 02 07:59:34.308379 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.308352 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pmvw6\"" Nov 02 07:59:34.316243 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.316212 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n642\" (UniqueName: \"kubernetes.io/projected/75ebee78-8815-4006-9862-36f3747cc557-kube-api-access-6n642\") pod \"test-curl-7352-76e02c\" (UID: \"75ebee78-8815-4006-9862-36f3747cc557\") " pod="opendatahub/test-curl-7352-76e02c" Nov 02 07:59:34.417288 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.417251 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n642\" (UniqueName: \"kubernetes.io/projected/75ebee78-8815-4006-9862-36f3747cc557-kube-api-access-6n642\") pod \"test-curl-7352-76e02c\" (UID: \"75ebee78-8815-4006-9862-36f3747cc557\") " pod="opendatahub/test-curl-7352-76e02c" Nov 02 07:59:34.451788 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.451762 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n642\" (UniqueName: \"kubernetes.io/projected/75ebee78-8815-4006-9862-36f3747cc557-kube-api-access-6n642\") pod \"test-curl-7352-76e02c\" (UID: \"75ebee78-8815-4006-9862-36f3747cc557\") " pod="opendatahub/test-curl-7352-76e02c" Nov 02 07:59:34.592649 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.592572 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-76e02c" Nov 02 07:59:34.892474 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:34.892445 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7352-76e02c"] Nov 02 07:59:34.894596 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:59:34.894570 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75ebee78_8815_4006_9862_36f3747cc557.slice/crio-34492725de7d3f8cbb0b09925cc3b2442c838e85136f75aeceafb7b47cf39459 WatchSource:0}: Error finding container 34492725de7d3f8cbb0b09925cc3b2442c838e85136f75aeceafb7b47cf39459: Status 404 returned error can't find the container with id 34492725de7d3f8cbb0b09925cc3b2442c838e85136f75aeceafb7b47cf39459 Nov 02 07:59:35.107995 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:35.107945 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-76e02c" event={"ID":"75ebee78-8815-4006-9862-36f3747cc557","Type":"ContainerStarted","Data":"7a37802e41768a0bd1c37c9b7ce7f1fe9ad10446c56f2fd3c4c6218b396d85a5"} Nov 02 07:59:35.108232 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:35.108001 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7352-76e02c" event={"ID":"75ebee78-8815-4006-9862-36f3747cc557","Type":"ContainerStarted","Data":"34492725de7d3f8cbb0b09925cc3b2442c838e85136f75aeceafb7b47cf39459"} Nov 02 07:59:35.118850 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:35.118822 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:35.146576 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:35.146465 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7352-76e02c" podStartSLOduration=1.146447144 podStartE2EDuration="1.146447144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:35.146415917 +0000 UTC m=+2075.444696743" watchObservedRunningTime="2026-11-02 07:59:35.146447144 +0000 UTC m=+2075.444727968" Nov 02 07:59:35.825524 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:35.825487 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:36.008471 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:36.008434 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7352-76e02c"] Nov 02 07:59:36.020650 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:36.020621 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7352-76e02c"] Nov 02 07:59:36.066503 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:36.066466 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:36.713827 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:36.713794 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:37.119701 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:37.119660 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7352-76e02c" podUID="75ebee78-8815-4006-9862-36f3747cc557" containerName="test-curl-7352-76e02c" containerID="cri-o://7a37802e41768a0bd1c37c9b7ce7f1fe9ad10446c56f2fd3c4c6218b396d85a5" gracePeriod=2 Nov 02 07:59:37.133639 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:37.133607 5899 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7352-76e02c\" is forbidden: User \"system:node:ip-10-0-135-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-6.ec2.internal' and this object" podUID="75ebee78-8815-4006-9862-36f3747cc557" pod="opendatahub/test-curl-7352-76e02c" Nov 02 07:59:37.138854 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:37.138832 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:37.364172 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:37.364138 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:37.836055 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:37.836020 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:38.405819 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:38.405784 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:38.779863 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:38.779836 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:39.381744 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:39.381546 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-76e02c" Nov 02 07:59:39.485148 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:39.485116 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ebee78-8815-4006-9862-36f3747cc557-kube-api-access-6n642\" (UniqueName: \"kubernetes.io/projected/75ebee78-8815-4006-9862-36f3747cc557-kube-api-access-6n642\") pod \"75ebee78-8815-4006-9862-36f3747cc557\" (UID: \"75ebee78-8815-4006-9862-36f3747cc557\") " Nov 02 07:59:39.487394 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:39.487366 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ebee78-8815-4006-9862-36f3747cc557-kube-api-access-6n642" pod "75ebee78-8815-4006-9862-36f3747cc557" (UID: "75ebee78-8815-4006-9862-36f3747cc557"). InnerVolumeSpecName "kube-api-access-6n642". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:59:39.553203 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:39.552135 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:39.587236 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:39.587196 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n642\" (UniqueName: \"kubernetes.io/projected/75ebee78-8815-4006-9862-36f3747cc557-kube-api-access-6n642\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 07:59:39.820607 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:39.820570 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ebee78-8815-4006-9862-36f3747cc557" path="/var/lib/kubelet/pods/75ebee78-8815-4006-9862-36f3747cc557/volumes" Nov 02 07:59:40.139523 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:40.139442 5899 generic.go:356] "Generic (PLEG): container finished" podID="75ebee78-8815-4006-9862-36f3747cc557" containerID="7a37802e41768a0bd1c37c9b7ce7f1fe9ad10446c56f2fd3c4c6218b396d85a5" exitCode=137 Nov 02 07:59:40.139523 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:40.139483 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7352-76e02c" Nov 02 07:59:40.139523 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:40.139507 5899 scope.go:122] "RemoveContainer" containerID="7a37802e41768a0bd1c37c9b7ce7f1fe9ad10446c56f2fd3c4c6218b396d85a5" Nov 02 07:59:40.148507 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:40.148492 5899 scope.go:122] "RemoveContainer" containerID="7a37802e41768a0bd1c37c9b7ce7f1fe9ad10446c56f2fd3c4c6218b396d85a5" Nov 02 07:59:40.148755 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:40.148738 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a37802e41768a0bd1c37c9b7ce7f1fe9ad10446c56f2fd3c4c6218b396d85a5\": container with ID starting with 7a37802e41768a0bd1c37c9b7ce7f1fe9ad10446c56f2fd3c4c6218b396d85a5 not found: ID does not exist" containerID="7a37802e41768a0bd1c37c9b7ce7f1fe9ad10446c56f2fd3c4c6218b396d85a5" Nov 02 07:59:40.148804 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:40.148764 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a37802e41768a0bd1c37c9b7ce7f1fe9ad10446c56f2fd3c4c6218b396d85a5"} err="failed to get container status \"7a37802e41768a0bd1c37c9b7ce7f1fe9ad10446c56f2fd3c4c6218b396d85a5\": rpc error: code = NotFound desc = could not find container \"7a37802e41768a0bd1c37c9b7ce7f1fe9ad10446c56f2fd3c4c6218b396d85a5\": container with ID starting with 7a37802e41768a0bd1c37c9b7ce7f1fe9ad10446c56f2fd3c4c6218b396d85a5 not found: ID does not exist" Nov 02 07:59:40.634132 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:40.634099 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:40.861569 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:40.861493 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:41.031874 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:41.031837 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:41.873565 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:41.873518 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:43.141116 ip-10-0-135-6 kubenswrapper[5899]: E1102 07:59:43.141079 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 07:59:43.794143 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:43.794112 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m"] Nov 02 07:59:43.794660 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:43.794640 5899 cpu_manager.go:414] "containerMap: removing container" podUID="75ebee78-8815-4006-9862-36f3747cc557" containerName="test-curl-7352-76e02c" Nov 02 07:59:43.794710 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:43.794662 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ebee78-8815-4006-9862-36f3747cc557" containerName="test-curl-7352-76e02c" Nov 02 07:59:43.794805 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:43.794795 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ebee78-8815-4006-9862-36f3747cc557" containerName="test-curl-7352-76e02c" Nov 02 07:59:43.797738 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:43.797719 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:43.813880 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:43.813854 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c712b1-serving-cert\"" Nov 02 07:59:43.833512 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:43.833482 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m"] Nov 02 07:59:43.923984 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:43.923949 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:43.951786 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:43.951752 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:43.951949 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:43.951825 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ee734d-db5b-4d7f-aff3-044bb4802ba1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:43.951949 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:43.951906 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-metrics-certs\") pod \"maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:43.952102 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:43.951983 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgzsr\" (UniqueName: \"kubernetes.io/projected/73ee734d-db5b-4d7f-aff3-044bb4802ba1-kube-api-access-kgzsr\") pod \"maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:44.053910 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.053823 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ee734d-db5b-4d7f-aff3-044bb4802ba1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:44.053910 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.053862 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-metrics-certs\") pod \"maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:44.053910 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.053889 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgzsr\" (UniqueName: \"kubernetes.io/projected/73ee734d-db5b-4d7f-aff3-044bb4802ba1-kube-api-access-kgzsr\") pod \"maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:44.054227 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.054023 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:44.054609 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.054583 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ee734d-db5b-4d7f-aff3-044bb4802ba1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:44.056578 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.056551 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:44.056666 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.056586 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-metrics-certs\") pod \"maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:44.078006 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.077978 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgzsr\" (UniqueName: \"kubernetes.io/projected/73ee734d-db5b-4d7f-aff3-044bb4802ba1-kube-api-access-kgzsr\") pod \"maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:44.114969 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.114937 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:44.149583 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.149550 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:44.334277 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.334191 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:44.430246 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.430215 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m"] Nov 02 07:59:44.431816 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:59:44.431793 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ee734d_db5b_4d7f_aff3_044bb4802ba1.slice/crio-ea93cfa0d793d045107b0fdba4cc31e4503544c5ab585629780b044b14193ea9 WatchSource:0}: Error finding container ea93cfa0d793d045107b0fdba4cc31e4503544c5ab585629780b044b14193ea9: Status 404 returned error can't find the container with id ea93cfa0d793d045107b0fdba4cc31e4503544c5ab585629780b044b14193ea9 Nov 02 07:59:44.433263 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.433241 5899 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 07:59:44.781700 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.781668 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c"] Nov 02 07:59:44.786904 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.786882 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.818466 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.818435 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-7f38f1\"/\"kube-root-ca.crt\"" Nov 02 07:59:44.818634 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.818554 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-7f38f1\"/\"noten-model-7f38f1-kserve-self-signed-certs\"" Nov 02 07:59:44.818856 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.818836 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-7f38f1\"/\"openshift-service-ca.crt\"" Nov 02 07:59:44.818976 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.818867 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-7f38f1\"/\"default-dockercfg-v94tc\"" Nov 02 07:59:44.825906 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.825880 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c"] Nov 02 07:59:44.864371 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.864343 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df9cec19-431f-4e04-b075-25c4d3759259-tls-certs\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.864486 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.864405 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-model-cache\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.864616 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.864592 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-home\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.864761 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.864741 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx79b\" (UniqueName: \"kubernetes.io/projected/df9cec19-431f-4e04-b075-25c4d3759259-kube-api-access-fx79b\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.864826 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.864793 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-dshm\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.864886 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.864848 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-tmp-dir\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.889950 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.889918 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:44.965866 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.965831 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-tmp-dir\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.966032 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.965941 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df9cec19-431f-4e04-b075-25c4d3759259-tls-certs\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.966032 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.966014 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-model-cache\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.966221 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.966190 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-tmp-dir\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.966221 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.966153 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-home\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.967101 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.966706 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-model-cache\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.967101 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.966811 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-home\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.967101 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.967071 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx79b\" (UniqueName: \"kubernetes.io/projected/df9cec19-431f-4e04-b075-25c4d3759259-kube-api-access-fx79b\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.967945 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.967904 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-dshm\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.970637 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.970103 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df9cec19-431f-4e04-b075-25c4d3759259-tls-certs\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.976735 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.971363 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-dshm\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:44.997875 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:44.997838 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx79b\" (UniqueName: \"kubernetes.io/projected/df9cec19-431f-4e04-b075-25c4d3759259-kube-api-access-fx79b\") pod \"noten-model-7f38f1-kserve-754cbb6b77-4s65c\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:45.101926 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:45.101843 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:45.220257 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:45.220223 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" event={"ID":"73ee734d-db5b-4d7f-aff3-044bb4802ba1","Type":"ContainerStarted","Data":"c39e7d1872f13925c5c7dd2e5abff161445f32dc93b2bdc8a62cf590274e9bc4"} Nov 02 07:59:45.220772 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:45.220754 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" event={"ID":"73ee734d-db5b-4d7f-aff3-044bb4802ba1","Type":"ContainerStarted","Data":"ea93cfa0d793d045107b0fdba4cc31e4503544c5ab585629780b044b14193ea9"} Nov 02 07:59:45.221738 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:45.221723 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:45.272234 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:45.269578 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" podStartSLOduration=1.8435453050000001 podStartE2EDuration="2.269563293s" totalImagesPullingTime="426.017988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:45.268869384 +0000 UTC m=+2085.567150212" watchObservedRunningTime="2026-11-02 07:59:45.269563293 +0000 UTC m=+2085.567844117" Nov 02 07:59:45.533558 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:45.533521 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c"] Nov 02 07:59:45.540024 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:59:45.539992 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf9cec19_431f_4e04_b075_25c4d3759259.slice/crio-9c7b16b5048d72e47c3d7af61cf9199f3fe04e4b45edae4d0ed91ba783020205 WatchSource:0}: Error finding container 9c7b16b5048d72e47c3d7af61cf9199f3fe04e4b45edae4d0ed91ba783020205: Status 404 returned error can't find the container with id 9c7b16b5048d72e47c3d7af61cf9199f3fe04e4b45edae4d0ed91ba783020205 Nov 02 07:59:46.230516 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:46.230471 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" event={"ID":"df9cec19-431f-4e04-b075-25c4d3759259","Type":"ContainerStarted","Data":"ce945a3a58a4d567c41b630ebd4b968e355c9d003840b3519056a05d72d382d5"} Nov 02 07:59:46.230919 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:46.230523 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" event={"ID":"df9cec19-431f-4e04-b075-25c4d3759259","Type":"ContainerStarted","Data":"9c7b16b5048d72e47c3d7af61cf9199f3fe04e4b45edae4d0ed91ba783020205"} Nov 02 07:59:46.254475 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:46.254443 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:46.278129 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:46.278079 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" podStartSLOduration=2.278064488 podStartE2EDuration="2.278064488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:46.274980724 +0000 UTC m=+2086.573261549" watchObservedRunningTime="2026-11-02 07:59:46.278064488 +0000 UTC m=+2086.576345334" Nov 02 07:59:47.102133 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:47.102097 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:48.428182 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:48.428125 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:50.302531 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:50.302487 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:52.133619 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:52.128683 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:52.241885 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:52.241854 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 07:59:54.013223 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:54.010556 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:55.102946 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:55.102916 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:55.116173 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:55.116141 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:55.295052 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:55.295022 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 07:59:56.025548 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:56.025514 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:58.175481 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.175447 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz"] Nov 02 07:59:58.178909 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.178886 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.194092 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.194073 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-696c7a-serving-cert\"" Nov 02 07:59:58.207891 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.207863 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz"] Nov 02 07:59:58.288665 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.288631 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-metrics-certs\") pod \"maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.288843 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.288684 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgb55\" (UniqueName: \"kubernetes.io/projected/8f09bad9-8a3f-45ba-93c3-3bc09e153761-kube-api-access-kgb55\") pod \"maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.288843 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.288746 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-maas-api-tls\") pod \"maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.288843 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.288804 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f09bad9-8a3f-45ba-93c3-3bc09e153761-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.374724 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.374693 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:58.390296 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.390268 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-metrics-certs\") pod \"maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.390446 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.390321 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgb55\" (UniqueName: \"kubernetes.io/projected/8f09bad9-8a3f-45ba-93c3-3bc09e153761-kube-api-access-kgb55\") pod \"maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.390446 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.390348 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-maas-api-tls\") pod \"maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.390446 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.390394 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f09bad9-8a3f-45ba-93c3-3bc09e153761-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.391069 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.391049 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f09bad9-8a3f-45ba-93c3-3bc09e153761-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.392943 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.392912 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-maas-api-tls\") pod \"maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.393032 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.392922 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-metrics-certs\") pod \"maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.415547 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.415520 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgb55\" (UniqueName: \"kubernetes.io/projected/8f09bad9-8a3f-45ba-93c3-3bc09e153761-kube-api-access-kgb55\") pod \"maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.495682 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.495642 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:58.522713 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.522684 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 07:59:58.803632 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:58.803602 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz"] Nov 02 07:59:58.805504 ip-10-0-135-6 kubenswrapper[5899]: W1102 07:59:58.805478 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f09bad9_8a3f_45ba_93c3_3bc09e153761.slice/crio-565ca3aeac7e1cd4d739c4176fd7084f71e74491ed22d63681f5355f475ad098 WatchSource:0}: Error finding container 565ca3aeac7e1cd4d739c4176fd7084f71e74491ed22d63681f5355f475ad098: Status 404 returned error can't find the container with id 565ca3aeac7e1cd4d739c4176fd7084f71e74491ed22d63681f5355f475ad098 Nov 02 07:59:59.312197 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:59.312137 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" event={"ID":"8f09bad9-8a3f-45ba-93c3-3bc09e153761","Type":"ContainerStarted","Data":"4342aab66622b90378de9be23b1ce2bf1ddca7664c08f0b3c2bcd1d5efb41a6a"} Nov 02 07:59:59.312197 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:59.312201 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" event={"ID":"8f09bad9-8a3f-45ba-93c3-3bc09e153761","Type":"ContainerStarted","Data":"565ca3aeac7e1cd4d739c4176fd7084f71e74491ed22d63681f5355f475ad098"} Nov 02 07:59:59.312738 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:59.312233 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 07:59:59.359802 ip-10-0-135-6 kubenswrapper[5899]: I1102 07:59:59.359733 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" podStartSLOduration=1.019965723 podStartE2EDuration="1.359713591s" totalImagesPullingTime="339.747868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:59.356813434 +0000 UTC m=+2099.655094259" watchObservedRunningTime="2026-11-02 07:59:59.359713591 +0000 UTC m=+2099.657994420" Nov 02 08:00:00.245899 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.245860 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2"] Nov 02 08:00:00.250293 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.250264 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" Nov 02 08:00:00.285557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.285515 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\"" Nov 02 08:00:00.286286 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.286217 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2"] Nov 02 08:00:00.347185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.328067 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s"] Nov 02 08:00:00.347185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.342398 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s"] Nov 02 08:00:00.347185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.342557 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" Nov 02 08:00:00.431419 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.431391 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w29nm\" (UniqueName: \"kubernetes.io/projected/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-kube-api-access-w29nm\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2\" (UID: \"dfd25a8a-cdc8-415b-b736-0fa59bd3719e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" Nov 02 08:00:00.431553 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.431530 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2\" (UID: \"dfd25a8a-cdc8-415b-b736-0fa59bd3719e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" Nov 02 08:00:00.533185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.533089 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f58v5\" (UniqueName: \"kubernetes.io/projected/be1305b8-1ae0-44f4-a860-b19b01e5449e-kube-api-access-f58v5\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s\" (UID: \"be1305b8-1ae0-44f4-a860-b19b01e5449e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" Nov 02 08:00:00.533394 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.533369 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be1305b8-1ae0-44f4-a860-b19b01e5449e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s\" (UID: \"be1305b8-1ae0-44f4-a860-b19b01e5449e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" Nov 02 08:00:00.533589 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.533573 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w29nm\" (UniqueName: \"kubernetes.io/projected/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-kube-api-access-w29nm\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2\" (UID: \"dfd25a8a-cdc8-415b-b736-0fa59bd3719e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" Nov 02 08:00:00.533672 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.533656 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2\" (UID: \"dfd25a8a-cdc8-415b-b736-0fa59bd3719e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" Nov 02 08:00:00.534120 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.534103 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2\" (UID: \"dfd25a8a-cdc8-415b-b736-0fa59bd3719e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" Nov 02 08:00:00.556542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.556507 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w29nm\" (UniqueName: \"kubernetes.io/projected/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-kube-api-access-w29nm\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2\" (UID: \"dfd25a8a-cdc8-415b-b736-0fa59bd3719e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" Nov 02 08:00:00.576329 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.576300 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" Nov 02 08:00:00.634898 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.634869 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f58v5\" (UniqueName: \"kubernetes.io/projected/be1305b8-1ae0-44f4-a860-b19b01e5449e-kube-api-access-f58v5\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s\" (UID: \"be1305b8-1ae0-44f4-a860-b19b01e5449e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" Nov 02 08:00:00.635045 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.634987 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be1305b8-1ae0-44f4-a860-b19b01e5449e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s\" (UID: \"be1305b8-1ae0-44f4-a860-b19b01e5449e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" Nov 02 08:00:00.636091 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.636067 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be1305b8-1ae0-44f4-a860-b19b01e5449e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s\" (UID: \"be1305b8-1ae0-44f4-a860-b19b01e5449e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" Nov 02 08:00:00.658076 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.658040 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f58v5\" (UniqueName: \"kubernetes.io/projected/be1305b8-1ae0-44f4-a860-b19b01e5449e-kube-api-access-f58v5\") pod \"maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s\" (UID: \"be1305b8-1ae0-44f4-a860-b19b01e5449e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" Nov 02 08:00:00.672418 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.672386 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" Nov 02 08:00:00.879081 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.879051 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2"] Nov 02 08:00:00.880714 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:00:00.880685 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd25a8a_cdc8_415b_b736_0fa59bd3719e.slice/crio-2e1677e9b0dbd53968a8d5265c80ff768ebb0b5a1cf871075893341b26098cbf WatchSource:0}: Error finding container 2e1677e9b0dbd53968a8d5265c80ff768ebb0b5a1cf871075893341b26098cbf: Status 404 returned error can't find the container with id 2e1677e9b0dbd53968a8d5265c80ff768ebb0b5a1cf871075893341b26098cbf Nov 02 08:00:00.980706 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:00.980682 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s"] Nov 02 08:00:00.982387 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:00:00.982358 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1305b8_1ae0_44f4_a860_b19b01e5449e.slice/crio-f2de36c104a53e4b6391715a6c3e427a3f3e37a086ecefc3acc924aeb6604ca4 WatchSource:0}: Error finding container f2de36c104a53e4b6391715a6c3e427a3f3e37a086ecefc3acc924aeb6604ca4: Status 404 returned error can't find the container with id f2de36c104a53e4b6391715a6c3e427a3f3e37a086ecefc3acc924aeb6604ca4 Nov 02 08:00:01.362731 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:01.362671 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" event={"ID":"dfd25a8a-cdc8-415b-b736-0fa59bd3719e","Type":"ContainerStarted","Data":"8401c7bc7a266f387c07dda33842417259595a5cb524e1c3f54af0e2d42eef48"} Nov 02 08:00:01.362731 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:01.362737 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" event={"ID":"dfd25a8a-cdc8-415b-b736-0fa59bd3719e","Type":"ContainerStarted","Data":"2e1677e9b0dbd53968a8d5265c80ff768ebb0b5a1cf871075893341b26098cbf"} Nov 02 08:00:01.385187 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:01.382964 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:01.385187 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:01.383509 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" event={"ID":"be1305b8-1ae0-44f4-a860-b19b01e5449e","Type":"ContainerStarted","Data":"9f5cd1ab9534c6ecb60aa0a6dd67c9f0423b43fb58eb032e480c6220ba1801e3"} Nov 02 08:00:01.385187 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:01.383567 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" event={"ID":"be1305b8-1ae0-44f4-a860-b19b01e5449e","Type":"ContainerStarted","Data":"f2de36c104a53e4b6391715a6c3e427a3f3e37a086ecefc3acc924aeb6604ca4"} Nov 02 08:00:01.407626 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:01.407572 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" podStartSLOduration=1.40755579 podStartE2EDuration="1.40755579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:01.405430382 +0000 UTC m=+2101.703711208" watchObservedRunningTime="2026-11-02 08:00:01.40755579 +0000 UTC m=+2101.705836615" Nov 02 08:00:01.459609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:01.459234 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" podStartSLOduration=1.459216186 podStartE2EDuration="1.459216186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:01.44734427 +0000 UTC m=+2101.745625094" watchObservedRunningTime="2026-11-02 08:00:01.459216186 +0000 UTC m=+2101.757497013" Nov 02 08:00:02.391514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:02.391475 5899 generic.go:356] "Generic (PLEG): container finished" podID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" containerID="8401c7bc7a266f387c07dda33842417259595a5cb524e1c3f54af0e2d42eef48" exitCode=60 Nov 02 08:00:02.391944 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:02.391525 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" event={"ID":"dfd25a8a-cdc8-415b-b736-0fa59bd3719e","Type":"ContainerDied","Data":"8401c7bc7a266f387c07dda33842417259595a5cb524e1c3f54af0e2d42eef48"} Nov 02 08:00:02.391944 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:02.391891 5899 scope.go:122] "RemoveContainer" containerID="8401c7bc7a266f387c07dda33842417259595a5cb524e1c3f54af0e2d42eef48" Nov 02 08:00:02.393402 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:02.393380 5899 generic.go:356] "Generic (PLEG): container finished" podID="be1305b8-1ae0-44f4-a860-b19b01e5449e" containerID="9f5cd1ab9534c6ecb60aa0a6dd67c9f0423b43fb58eb032e480c6220ba1801e3" exitCode=7 Nov 02 08:00:02.393498 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:02.393430 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" event={"ID":"be1305b8-1ae0-44f4-a860-b19b01e5449e","Type":"ContainerDied","Data":"9f5cd1ab9534c6ecb60aa0a6dd67c9f0423b43fb58eb032e480c6220ba1801e3"} Nov 02 08:00:02.393682 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:02.393668 5899 scope.go:122] "RemoveContainer" containerID="9f5cd1ab9534c6ecb60aa0a6dd67c9f0423b43fb58eb032e480c6220ba1801e3" Nov 02 08:00:03.302998 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:03.302970 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:03.401817 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:03.401780 5899 generic.go:356] "Generic (PLEG): container finished" podID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" containerID="84d41697f1182c49849b86a116256199bbea3870d03e5c2a9b972632a43153b9" exitCode=60 Nov 02 08:00:03.402295 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:03.401868 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" event={"ID":"dfd25a8a-cdc8-415b-b736-0fa59bd3719e","Type":"ContainerDied","Data":"84d41697f1182c49849b86a116256199bbea3870d03e5c2a9b972632a43153b9"} Nov 02 08:00:03.402295 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:03.401930 5899 scope.go:122] "RemoveContainer" containerID="8401c7bc7a266f387c07dda33842417259595a5cb524e1c3f54af0e2d42eef48" Nov 02 08:00:03.402295 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:03.402211 5899 scope.go:122] "RemoveContainer" containerID="84d41697f1182c49849b86a116256199bbea3870d03e5c2a9b972632a43153b9" Nov 02 08:00:03.402554 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:00:03.402530 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2_odh-ai-gateway-infra(dfd25a8a-cdc8-415b-b736-0fa59bd3719e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" podUID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" Nov 02 08:00:03.404035 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:03.404013 5899 generic.go:356] "Generic (PLEG): container finished" podID="be1305b8-1ae0-44f4-a860-b19b01e5449e" containerID="75f8337e9e0fef96e34ac07156e50417742f879540c915e6590632eab6833b31" exitCode=7 Nov 02 08:00:03.404119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:03.404095 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" event={"ID":"be1305b8-1ae0-44f4-a860-b19b01e5449e","Type":"ContainerDied","Data":"75f8337e9e0fef96e34ac07156e50417742f879540c915e6590632eab6833b31"} Nov 02 08:00:03.404375 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:03.404362 5899 scope.go:122] "RemoveContainer" containerID="75f8337e9e0fef96e34ac07156e50417742f879540c915e6590632eab6833b31" Nov 02 08:00:03.404629 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:00:03.404607 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s_odh-ai-gateway-infra(be1305b8-1ae0-44f4-a860-b19b01e5449e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" podUID="be1305b8-1ae0-44f4-a860-b19b01e5449e" Nov 02 08:00:03.414432 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:03.414413 5899 scope.go:122] "RemoveContainer" containerID="9f5cd1ab9534c6ecb60aa0a6dd67c9f0423b43fb58eb032e480c6220ba1801e3" Nov 02 08:00:04.030345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:04.030311 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq"] Nov 02 08:00:04.030584 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:04.030556 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" podUID="c36051c8-200c-49f8-b02b-652bc857cedc" containerName="istio-proxy" containerID="cri-o://563ec7325e06a124cd56b7707a7c0a0965788eadac48f930989275c19e1065e1" gracePeriod=30 Nov 02 08:00:04.416282 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:04.416206 5899 scope.go:122] "RemoveContainer" containerID="84d41697f1182c49849b86a116256199bbea3870d03e5c2a9b972632a43153b9" Nov 02 08:00:04.416722 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:00:04.416573 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2_odh-ai-gateway-infra(dfd25a8a-cdc8-415b-b736-0fa59bd3719e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" podUID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" Nov 02 08:00:04.417831 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:04.417807 5899 scope.go:122] "RemoveContainer" containerID="75f8337e9e0fef96e34ac07156e50417742f879540c915e6590632eab6833b31" Nov 02 08:00:04.418101 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:00:04.418081 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s_odh-ai-gateway-infra(be1305b8-1ae0-44f4-a860-b19b01e5449e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" podUID="be1305b8-1ae0-44f4-a860-b19b01e5449e" Nov 02 08:00:04.973873 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:04.973836 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:05.278626 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:05.278595 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:05.357031 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:05.356996 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 08:00:06.860310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:06.860273 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c"] Nov 02 08:00:06.962206 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:06.961206 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:07.140234 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:07.140133 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:07.307061 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:07.306259 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:07.311637 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:07.311604 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng"] Nov 02 08:00:07.587406 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:07.587355 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:07.832739 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:07.832704 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:08.115673 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:08.115641 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:08.928344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:08.928302 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:00:08.928799 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:08.928358 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" podUID="c36051c8-200c-49f8-b02b-652bc857cedc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:00:09.308612 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.308585 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 08:00:09.362919 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.362889 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-credential-socket\") pod \"c36051c8-200c-49f8-b02b-652bc857cedc\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " Nov 02 08:00:09.363068 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.362964 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-socket\") pod \"c36051c8-200c-49f8-b02b-652bc857cedc\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " Nov 02 08:00:09.363068 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.362998 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c36051c8-200c-49f8-b02b-652bc857cedc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36051c8-200c-49f8-b02b-652bc857cedc-istio-podinfo\") pod \"c36051c8-200c-49f8-b02b-652bc857cedc\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " Nov 02 08:00:09.363068 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.363033 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-kube-api-access-qq7kh\" (UniqueName: \"kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-kube-api-access-qq7kh\") pod \"c36051c8-200c-49f8-b02b-652bc857cedc\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " Nov 02 08:00:09.363233 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.363129 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c36051c8-200c-49f8-b02b-652bc857cedc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36051c8-200c-49f8-b02b-652bc857cedc-istiod-ca-cert\") pod \"c36051c8-200c-49f8-b02b-652bc857cedc\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " Nov 02 08:00:09.363233 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.363185 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-envoy\") pod \"c36051c8-200c-49f8-b02b-652bc857cedc\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " Nov 02 08:00:09.363337 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.363239 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-socket" pod "c36051c8-200c-49f8-b02b-652bc857cedc" (UID: "c36051c8-200c-49f8-b02b-652bc857cedc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:09.363337 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.363260 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-credential-socket" pod "c36051c8-200c-49f8-b02b-652bc857cedc" (UID: "c36051c8-200c-49f8-b02b-652bc857cedc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:09.363337 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.363249 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-data\") pod \"c36051c8-200c-49f8-b02b-652bc857cedc\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " Nov 02 08:00:09.363480 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.363336 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-certs\") pod \"c36051c8-200c-49f8-b02b-652bc857cedc\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " Nov 02 08:00:09.363480 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.363431 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-istio-token\" (UniqueName: \"kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-istio-token\") pod \"c36051c8-200c-49f8-b02b-652bc857cedc\" (UID: \"c36051c8-200c-49f8-b02b-652bc857cedc\") " Nov 02 08:00:09.363664 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.363613 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-certs" pod "c36051c8-200c-49f8-b02b-652bc857cedc" (UID: "c36051c8-200c-49f8-b02b-652bc857cedc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:09.363664 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.363643 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-data" pod "c36051c8-200c-49f8-b02b-652bc857cedc" (UID: "c36051c8-200c-49f8-b02b-652bc857cedc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:09.363664 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.363653 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c36051c8-200c-49f8-b02b-652bc857cedc-istiod-ca-cert" pod "c36051c8-200c-49f8-b02b-652bc857cedc" (UID: "c36051c8-200c-49f8-b02b-652bc857cedc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:00:09.364030 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.364012 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:09.364107 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.364035 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36051c8-200c-49f8-b02b-652bc857cedc-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:09.364107 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.364047 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:09.364107 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.364059 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:09.364107 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.364071 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:09.365563 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.365528 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-kube-api-access-qq7kh" pod "c36051c8-200c-49f8-b02b-652bc857cedc" (UID: "c36051c8-200c-49f8-b02b-652bc857cedc"). InnerVolumeSpecName "kube-api-access-qq7kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:00:09.365686 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.365663 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-istio-token" pod "c36051c8-200c-49f8-b02b-652bc857cedc" (UID: "c36051c8-200c-49f8-b02b-652bc857cedc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:00:09.365842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.365820 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c36051c8-200c-49f8-b02b-652bc857cedc-istio-podinfo" pod "c36051c8-200c-49f8-b02b-652bc857cedc" (UID: "c36051c8-200c-49f8-b02b-652bc857cedc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:00:09.365951 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.365887 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-envoy" pod "c36051c8-200c-49f8-b02b-652bc857cedc" (UID: "c36051c8-200c-49f8-b02b-652bc857cedc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:09.464871 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.464796 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36051c8-200c-49f8-b02b-652bc857cedc-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:09.464871 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.464822 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:09.464871 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.464831 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36051c8-200c-49f8-b02b-652bc857cedc-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:09.464871 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.464841 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq7kh\" (UniqueName: \"kubernetes.io/projected/c36051c8-200c-49f8-b02b-652bc857cedc-kube-api-access-qq7kh\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:09.474821 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.473999 5899 generic.go:356] "Generic (PLEG): container finished" podID="c36051c8-200c-49f8-b02b-652bc857cedc" containerID="563ec7325e06a124cd56b7707a7c0a0965788eadac48f930989275c19e1065e1" exitCode=0 Nov 02 08:00:09.474821 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.474063 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" Nov 02 08:00:09.474821 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.474130 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" event={"ID":"c36051c8-200c-49f8-b02b-652bc857cedc","Type":"ContainerDied","Data":"563ec7325e06a124cd56b7707a7c0a0965788eadac48f930989275c19e1065e1"} Nov 02 08:00:09.474821 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.474211 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq" event={"ID":"c36051c8-200c-49f8-b02b-652bc857cedc","Type":"ContainerDied","Data":"fb893f3708aeb5b3750e7cdfc6609d00dac1a307abbab63f9ddcb4b43cc43247"} Nov 02 08:00:09.474821 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.474248 5899 scope.go:122] "RemoveContainer" containerID="563ec7325e06a124cd56b7707a7c0a0965788eadac48f930989275c19e1065e1" Nov 02 08:00:09.487966 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.487942 5899 scope.go:122] "RemoveContainer" containerID="563ec7325e06a124cd56b7707a7c0a0965788eadac48f930989275c19e1065e1" Nov 02 08:00:09.488300 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:00:09.488280 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563ec7325e06a124cd56b7707a7c0a0965788eadac48f930989275c19e1065e1\": container with ID starting with 563ec7325e06a124cd56b7707a7c0a0965788eadac48f930989275c19e1065e1 not found: ID does not exist" containerID="563ec7325e06a124cd56b7707a7c0a0965788eadac48f930989275c19e1065e1" Nov 02 08:00:09.488359 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.488310 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563ec7325e06a124cd56b7707a7c0a0965788eadac48f930989275c19e1065e1"} err="failed to get container status \"563ec7325e06a124cd56b7707a7c0a0965788eadac48f930989275c19e1065e1\": rpc error: code = NotFound desc = could not find container \"563ec7325e06a124cd56b7707a7c0a0965788eadac48f930989275c19e1065e1\": container with ID starting with 563ec7325e06a124cd56b7707a7c0a0965788eadac48f930989275c19e1065e1 not found: ID does not exist" Nov 02 08:00:09.525438 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.525404 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq"] Nov 02 08:00:09.531379 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.531353 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-9khqq"] Nov 02 08:00:09.638445 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.638408 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:09.828399 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:09.828366 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c36051c8-200c-49f8-b02b-652bc857cedc" path="/var/lib/kubelet/pods/c36051c8-200c-49f8-b02b-652bc857cedc/volumes" Nov 02 08:00:10.199525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:10.199436 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp"] Nov 02 08:00:10.199746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:10.199692 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" podUID="6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" containerName="istio-proxy" containerID="cri-o://ea4e86594e26b1767625bf95560905feea104ece72ad13d1d85d2857f7446b67" gracePeriod=30 Nov 02 08:00:11.410220 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:11.410186 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:12.617607 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:12.617570 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:13.136220 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:13.136188 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:13.140736 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:00:13.140709 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:00:14.844295 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:14.844257 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:15.348630 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.348591 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:15.372216 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.371641 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 08:00:15.372216 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.371699 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" podUID="6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 08:00:15.488258 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.488231 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 08:00:15.514395 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.514363 5899 generic.go:356] "Generic (PLEG): container finished" podID="6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" containerID="ea4e86594e26b1767625bf95560905feea104ece72ad13d1d85d2857f7446b67" exitCode=0 Nov 02 08:00:15.514567 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.514411 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" Nov 02 08:00:15.514567 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.514450 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" event={"ID":"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef","Type":"ContainerDied","Data":"ea4e86594e26b1767625bf95560905feea104ece72ad13d1d85d2857f7446b67"} Nov 02 08:00:15.514567 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.514503 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp" event={"ID":"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef","Type":"ContainerDied","Data":"0e88263f2c6c7128f93e42bc1b6cb1886d85b60b2ccbb11c11ec28ca9b41bdaf"} Nov 02 08:00:15.514567 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.514529 5899 scope.go:122] "RemoveContainer" containerID="ea4e86594e26b1767625bf95560905feea104ece72ad13d1d85d2857f7446b67" Nov 02 08:00:15.524409 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.524393 5899 scope.go:122] "RemoveContainer" containerID="ea4e86594e26b1767625bf95560905feea104ece72ad13d1d85d2857f7446b67" Nov 02 08:00:15.524710 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:00:15.524691 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4e86594e26b1767625bf95560905feea104ece72ad13d1d85d2857f7446b67\": container with ID starting with ea4e86594e26b1767625bf95560905feea104ece72ad13d1d85d2857f7446b67 not found: ID does not exist" containerID="ea4e86594e26b1767625bf95560905feea104ece72ad13d1d85d2857f7446b67" Nov 02 08:00:15.524797 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.524722 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4e86594e26b1767625bf95560905feea104ece72ad13d1d85d2857f7446b67"} err="failed to get container status \"ea4e86594e26b1767625bf95560905feea104ece72ad13d1d85d2857f7446b67\": rpc error: code = NotFound desc = could not find container \"ea4e86594e26b1767625bf95560905feea104ece72ad13d1d85d2857f7446b67\": container with ID starting with ea4e86594e26b1767625bf95560905feea104ece72ad13d1d85d2857f7446b67 not found: ID does not exist" Nov 02 08:00:15.542120 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542092 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-certs\") pod \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " Nov 02 08:00:15.542278 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542211 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-credential-socket\") pod \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " Nov 02 08:00:15.542278 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542244 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-envoy\") pod \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " Nov 02 08:00:15.542407 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542291 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istiod-ca-cert\") pod \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " Nov 02 08:00:15.542407 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542316 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-podinfo\") pod \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " Nov 02 08:00:15.542407 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542350 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-data\") pod \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " Nov 02 08:00:15.542566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542413 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-certs" pod "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" (UID: "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:15.542566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542436 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-token\" (UniqueName: \"kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-token\") pod \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " Nov 02 08:00:15.542566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542498 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-kube-api-access-56xvv\" (UniqueName: \"kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-kube-api-access-56xvv\") pod \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " Nov 02 08:00:15.542566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542523 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-credential-socket" pod "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" (UID: "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:15.542566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542554 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-socket\") pod \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\" (UID: \"6f72f7a8-6385-4190-9a7c-beecdd3ef7ef\") " Nov 02 08:00:15.542824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542796 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-data" pod "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" (UID: "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:15.542887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542825 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istiod-ca-cert" pod "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" (UID: "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:00:15.542941 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.542913 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-socket" pod "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" (UID: "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:15.543137 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.543120 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:15.543235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.543140 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:15.543235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.543149 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:15.543235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.543176 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:15.543235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.543190 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:15.544643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.544620 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-kube-api-access-56xvv" pod "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" (UID: "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef"). InnerVolumeSpecName "kube-api-access-56xvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:00:15.544732 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.544652 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-podinfo" pod "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" (UID: "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:00:15.544894 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.544873 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-token" pod "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" (UID: "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:00:15.544945 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.544892 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-envoy" pod "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" (UID: "6f72f7a8-6385-4190-9a7c-beecdd3ef7ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:15.644652 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.644613 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:15.644652 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.644641 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:15.644652 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.644650 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:15.644652 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.644660 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56xvv\" (UniqueName: \"kubernetes.io/projected/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef-kube-api-access-56xvv\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:15.863109 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.863071 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp"] Nov 02 08:00:15.874363 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:15.874337 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-b29lp"] Nov 02 08:00:16.763304 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:16.763272 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:16.812880 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:16.812851 5899 scope.go:122] "RemoveContainer" containerID="75f8337e9e0fef96e34ac07156e50417742f879540c915e6590632eab6833b31" Nov 02 08:00:17.094857 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:17.094821 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:17.477261 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:17.476063 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:17.531302 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:17.531262 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" event={"ID":"be1305b8-1ae0-44f4-a860-b19b01e5449e","Type":"ContainerStarted","Data":"8e48a158c47815cec9476d17f94fffb99ad4bbf29e7017f551094d72bcfb0214"} Nov 02 08:00:17.812739 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:17.812656 5899 scope.go:122] "RemoveContainer" containerID="84d41697f1182c49849b86a116256199bbea3870d03e5c2a9b972632a43153b9" Nov 02 08:00:17.824105 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:17.824069 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" path="/var/lib/kubelet/pods/6f72f7a8-6385-4190-9a7c-beecdd3ef7ef/volumes" Nov 02 08:00:17.889070 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:17.889040 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:18.539823 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:18.539786 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" event={"ID":"dfd25a8a-cdc8-415b-b736-0fa59bd3719e","Type":"ContainerStarted","Data":"2053984f14c252cac8a290e7ab74161aab85a01197bd39ae807dc9ef57b8b3d0"} Nov 02 08:00:18.541869 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:18.541845 5899 generic.go:356] "Generic (PLEG): container finished" podID="be1305b8-1ae0-44f4-a860-b19b01e5449e" containerID="8e48a158c47815cec9476d17f94fffb99ad4bbf29e7017f551094d72bcfb0214" exitCode=60 Nov 02 08:00:18.542015 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:18.541913 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" event={"ID":"be1305b8-1ae0-44f4-a860-b19b01e5449e","Type":"ContainerDied","Data":"8e48a158c47815cec9476d17f94fffb99ad4bbf29e7017f551094d72bcfb0214"} Nov 02 08:00:18.542015 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:18.541968 5899 scope.go:122] "RemoveContainer" containerID="75f8337e9e0fef96e34ac07156e50417742f879540c915e6590632eab6833b31" Nov 02 08:00:18.542290 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:18.542267 5899 scope.go:122] "RemoveContainer" containerID="8e48a158c47815cec9476d17f94fffb99ad4bbf29e7017f551094d72bcfb0214" Nov 02 08:00:18.542587 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:00:18.542562 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s_odh-ai-gateway-infra(be1305b8-1ae0-44f4-a860-b19b01e5449e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" podUID="be1305b8-1ae0-44f4-a860-b19b01e5449e" Nov 02 08:00:18.595221 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:18.595184 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s"] Nov 02 08:00:19.166680 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:19.166639 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:19.552711 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:19.552684 5899 generic.go:356] "Generic (PLEG): container finished" podID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" containerID="2053984f14c252cac8a290e7ab74161aab85a01197bd39ae807dc9ef57b8b3d0" exitCode=60 Nov 02 08:00:19.553108 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:19.552759 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" event={"ID":"dfd25a8a-cdc8-415b-b736-0fa59bd3719e","Type":"ContainerDied","Data":"2053984f14c252cac8a290e7ab74161aab85a01197bd39ae807dc9ef57b8b3d0"} Nov 02 08:00:19.553108 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:19.552821 5899 scope.go:122] "RemoveContainer" containerID="84d41697f1182c49849b86a116256199bbea3870d03e5c2a9b972632a43153b9" Nov 02 08:00:19.553325 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:19.553309 5899 scope.go:122] "RemoveContainer" containerID="2053984f14c252cac8a290e7ab74161aab85a01197bd39ae807dc9ef57b8b3d0" Nov 02 08:00:19.553678 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:00:19.553651 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2_odh-ai-gateway-infra(dfd25a8a-cdc8-415b-b736-0fa59bd3719e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" podUID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" Nov 02 08:00:19.604640 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:19.604606 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2"] Nov 02 08:00:19.712018 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:19.711989 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" Nov 02 08:00:19.802699 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:19.802603 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1305b8-1ae0-44f4-a860-b19b01e5449e-kube-api-access-f58v5\" (UniqueName: \"kubernetes.io/projected/be1305b8-1ae0-44f4-a860-b19b01e5449e-kube-api-access-f58v5\") pod \"be1305b8-1ae0-44f4-a860-b19b01e5449e\" (UID: \"be1305b8-1ae0-44f4-a860-b19b01e5449e\") " Nov 02 08:00:19.802867 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:19.802712 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be1305b8-1ae0-44f4-a860-b19b01e5449e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be1305b8-1ae0-44f4-a860-b19b01e5449e-maas-api-service-ca\") pod \"be1305b8-1ae0-44f4-a860-b19b01e5449e\" (UID: \"be1305b8-1ae0-44f4-a860-b19b01e5449e\") " Nov 02 08:00:19.803185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:19.803140 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1305b8-1ae0-44f4-a860-b19b01e5449e-maas-api-service-ca" pod "be1305b8-1ae0-44f4-a860-b19b01e5449e" (UID: "be1305b8-1ae0-44f4-a860-b19b01e5449e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:00:19.803425 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:19.803406 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be1305b8-1ae0-44f4-a860-b19b01e5449e-maas-api-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:19.804799 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:19.804777 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1305b8-1ae0-44f4-a860-b19b01e5449e-kube-api-access-f58v5" pod "be1305b8-1ae0-44f4-a860-b19b01e5449e" (UID: "be1305b8-1ae0-44f4-a860-b19b01e5449e"). InnerVolumeSpecName "kube-api-access-f58v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:00:19.904858 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:19.904826 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f58v5\" (UniqueName: \"kubernetes.io/projected/be1305b8-1ae0-44f4-a860-b19b01e5449e-kube-api-access-f58v5\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:20.460523 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:20.460488 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:20.562623 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:20.562598 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" Nov 02 08:00:20.563074 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:20.562620 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s" event={"ID":"be1305b8-1ae0-44f4-a860-b19b01e5449e","Type":"ContainerDied","Data":"f2de36c104a53e4b6391715a6c3e427a3f3e37a086ecefc3acc924aeb6604ca4"} Nov 02 08:00:20.563074 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:20.562682 5899 scope.go:122] "RemoveContainer" containerID="8e48a158c47815cec9476d17f94fffb99ad4bbf29e7017f551094d72bcfb0214" Nov 02 08:00:20.670954 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:20.670868 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s"] Nov 02 08:00:20.674982 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:20.674951 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696c7a-29893440-kn48s"] Nov 02 08:00:20.722346 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:20.722276 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" Nov 02 08:00:20.814617 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:20.814583 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-kube-api-access-w29nm\" (UniqueName: \"kubernetes.io/projected/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-kube-api-access-w29nm\") pod \"dfd25a8a-cdc8-415b-b736-0fa59bd3719e\" (UID: \"dfd25a8a-cdc8-415b-b736-0fa59bd3719e\") " Nov 02 08:00:20.814773 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:20.814722 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-maas-api-service-ca\") pod \"dfd25a8a-cdc8-415b-b736-0fa59bd3719e\" (UID: \"dfd25a8a-cdc8-415b-b736-0fa59bd3719e\") " Nov 02 08:00:20.815196 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:20.815146 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-maas-api-service-ca" pod "dfd25a8a-cdc8-415b-b736-0fa59bd3719e" (UID: "dfd25a8a-cdc8-415b-b736-0fa59bd3719e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:00:20.815388 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:20.815362 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-maas-api-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:20.816838 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:20.816812 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-kube-api-access-w29nm" pod "dfd25a8a-cdc8-415b-b736-0fa59bd3719e" (UID: "dfd25a8a-cdc8-415b-b736-0fa59bd3719e"). InnerVolumeSpecName "kube-api-access-w29nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:00:20.916819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:20.916782 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w29nm\" (UniqueName: \"kubernetes.io/projected/dfd25a8a-cdc8-415b-b736-0fa59bd3719e-kube-api-access-w29nm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:21.332384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:21.330434 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:21.557130 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:21.557093 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:21.569479 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:21.569452 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" Nov 02 08:00:21.569955 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:21.569487 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2" event={"ID":"dfd25a8a-cdc8-415b-b736-0fa59bd3719e","Type":"ContainerDied","Data":"2e1677e9b0dbd53968a8d5265c80ff768ebb0b5a1cf871075893341b26098cbf"} Nov 02 08:00:21.569955 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:21.569581 5899 scope.go:122] "RemoveContainer" containerID="2053984f14c252cac8a290e7ab74161aab85a01197bd39ae807dc9ef57b8b3d0" Nov 02 08:00:21.629776 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:21.629691 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2"] Nov 02 08:00:21.641087 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:21.641059 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893440-4wsx2"] Nov 02 08:00:21.710591 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:21.710558 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:21.819761 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:21.819728 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be1305b8-1ae0-44f4-a860-b19b01e5449e" path="/var/lib/kubelet/pods/be1305b8-1ae0-44f4-a860-b19b01e5449e/volumes" Nov 02 08:00:21.820238 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:21.820223 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" path="/var/lib/kubelet/pods/dfd25a8a-cdc8-415b-b736-0fa59bd3719e/volumes" Nov 02 08:00:21.887509 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:21.887397 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" podUID="df9cec19-431f-4e04-b075-25c4d3759259" containerName="main" containerID="cri-o://ce945a3a58a4d567c41b630ebd4b968e355c9d003840b3519056a05d72d382d5" gracePeriod=45 Nov 02 08:00:21.965128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:21.965094 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:22.161986 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.161956 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 08:00:22.231578 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.231543 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df9cec19-431f-4e04-b075-25c4d3759259-tls-certs\" (UniqueName: \"kubernetes.io/secret/df9cec19-431f-4e04-b075-25c4d3759259-tls-certs\") pod \"df9cec19-431f-4e04-b075-25c4d3759259\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " Nov 02 08:00:22.231748 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.231634 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-home\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-home\") pod \"df9cec19-431f-4e04-b075-25c4d3759259\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " Nov 02 08:00:22.231748 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.231711 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-dshm\") pod \"df9cec19-431f-4e04-b075-25c4d3759259\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " Nov 02 08:00:22.231835 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.231808 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9cec19-431f-4e04-b075-25c4d3759259-kube-api-access-fx79b\" (UniqueName: \"kubernetes.io/projected/df9cec19-431f-4e04-b075-25c4d3759259-kube-api-access-fx79b\") pod \"df9cec19-431f-4e04-b075-25c4d3759259\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " Nov 02 08:00:22.231905 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.231886 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-model-cache\") pod \"df9cec19-431f-4e04-b075-25c4d3759259\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " Nov 02 08:00:22.231994 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.231923 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-tmp-dir\") pod \"df9cec19-431f-4e04-b075-25c4d3759259\" (UID: \"df9cec19-431f-4e04-b075-25c4d3759259\") " Nov 02 08:00:22.232751 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.232722 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-tmp-dir" pod "df9cec19-431f-4e04-b075-25c4d3759259" (UID: "df9cec19-431f-4e04-b075-25c4d3759259"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:22.232945 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.232921 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-model-cache" pod "df9cec19-431f-4e04-b075-25c4d3759259" (UID: "df9cec19-431f-4e04-b075-25c4d3759259"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:22.233132 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.233104 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-home" pod "df9cec19-431f-4e04-b075-25c4d3759259" (UID: "df9cec19-431f-4e04-b075-25c4d3759259"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:22.235614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.235568 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-dshm" pod "df9cec19-431f-4e04-b075-25c4d3759259" (UID: "df9cec19-431f-4e04-b075-25c4d3759259"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:22.235721 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.235633 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9cec19-431f-4e04-b075-25c4d3759259-tls-certs" pod "df9cec19-431f-4e04-b075-25c4d3759259" (UID: "df9cec19-431f-4e04-b075-25c4d3759259"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:00:22.235815 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.235791 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9cec19-431f-4e04-b075-25c4d3759259-kube-api-access-fx79b" pod "df9cec19-431f-4e04-b075-25c4d3759259" (UID: "df9cec19-431f-4e04-b075-25c4d3759259"). InnerVolumeSpecName "kube-api-access-fx79b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:00:22.337219 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.334304 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:22.337219 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.334336 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:22.337219 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.334351 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx79b\" (UniqueName: \"kubernetes.io/projected/df9cec19-431f-4e04-b075-25c4d3759259-kube-api-access-fx79b\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:22.337219 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.334367 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:22.337219 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.334384 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9cec19-431f-4e04-b075-25c4d3759259-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:22.337219 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.334396 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df9cec19-431f-4e04-b075-25c4d3759259-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:22.339100 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.339055 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" podUID="7db63785-0b2b-4516-b06d-3df792f2019f" containerName="main" containerID="cri-o://37e107ab874b2155c459cba959f1e2c29739c3065b0fc7f0189b6262a3e0430f" gracePeriod=45 Nov 02 08:00:22.493513 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.493443 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:22.581341 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.581310 5899 generic.go:356] "Generic (PLEG): container finished" podID="7db63785-0b2b-4516-b06d-3df792f2019f" containerID="37e107ab874b2155c459cba959f1e2c29739c3065b0fc7f0189b6262a3e0430f" exitCode=0 Nov 02 08:00:22.581732 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.581456 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" event={"ID":"7db63785-0b2b-4516-b06d-3df792f2019f","Type":"ContainerDied","Data":"37e107ab874b2155c459cba959f1e2c29739c3065b0fc7f0189b6262a3e0430f"} Nov 02 08:00:22.583403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.583324 5899 generic.go:356] "Generic (PLEG): container finished" podID="df9cec19-431f-4e04-b075-25c4d3759259" containerID="ce945a3a58a4d567c41b630ebd4b968e355c9d003840b3519056a05d72d382d5" exitCode=0 Nov 02 08:00:22.583545 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.583406 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" event={"ID":"df9cec19-431f-4e04-b075-25c4d3759259","Type":"ContainerDied","Data":"ce945a3a58a4d567c41b630ebd4b968e355c9d003840b3519056a05d72d382d5"} Nov 02 08:00:22.583545 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.583438 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" event={"ID":"df9cec19-431f-4e04-b075-25c4d3759259","Type":"ContainerDied","Data":"9c7b16b5048d72e47c3d7af61cf9199f3fe04e4b45edae4d0ed91ba783020205"} Nov 02 08:00:22.583545 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.583459 5899 scope.go:122] "RemoveContainer" containerID="ce945a3a58a4d567c41b630ebd4b968e355c9d003840b3519056a05d72d382d5" Nov 02 08:00:22.583702 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.583566 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c" Nov 02 08:00:22.597878 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.597853 5899 scope.go:122] "RemoveContainer" containerID="ce945a3a58a4d567c41b630ebd4b968e355c9d003840b3519056a05d72d382d5" Nov 02 08:00:22.598268 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:00:22.598231 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce945a3a58a4d567c41b630ebd4b968e355c9d003840b3519056a05d72d382d5\": container with ID starting with ce945a3a58a4d567c41b630ebd4b968e355c9d003840b3519056a05d72d382d5 not found: ID does not exist" containerID="ce945a3a58a4d567c41b630ebd4b968e355c9d003840b3519056a05d72d382d5" Nov 02 08:00:22.598363 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.598282 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce945a3a58a4d567c41b630ebd4b968e355c9d003840b3519056a05d72d382d5"} err="failed to get container status \"ce945a3a58a4d567c41b630ebd4b968e355c9d003840b3519056a05d72d382d5\": rpc error: code = NotFound desc = could not find container \"ce945a3a58a4d567c41b630ebd4b968e355c9d003840b3519056a05d72d382d5\": container with ID starting with ce945a3a58a4d567c41b630ebd4b968e355c9d003840b3519056a05d72d382d5 not found: ID does not exist" Nov 02 08:00:22.630489 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.630463 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 08:00:22.638899 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.638871 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c"] Nov 02 08:00:22.646877 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.646850 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-7f38f1/noten-model-7f38f1-kserve-754cbb6b77-4s65c"] Nov 02 08:00:22.739978 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.739946 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-tmp-dir\") pod \"7db63785-0b2b-4516-b06d-3df792f2019f\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " Nov 02 08:00:22.740205 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.740012 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-model-cache\") pod \"7db63785-0b2b-4516-b06d-3df792f2019f\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " Nov 02 08:00:22.740205 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.740084 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db63785-0b2b-4516-b06d-3df792f2019f-kube-api-access-ljl5t\" (UniqueName: \"kubernetes.io/projected/7db63785-0b2b-4516-b06d-3df792f2019f-kube-api-access-ljl5t\") pod \"7db63785-0b2b-4516-b06d-3df792f2019f\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " Nov 02 08:00:22.740338 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.740219 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-home\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-home\") pod \"7db63785-0b2b-4516-b06d-3df792f2019f\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " Nov 02 08:00:22.740338 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.740250 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7db63785-0b2b-4516-b06d-3df792f2019f-tls-certs\" (UniqueName: \"kubernetes.io/secret/7db63785-0b2b-4516-b06d-3df792f2019f-tls-certs\") pod \"7db63785-0b2b-4516-b06d-3df792f2019f\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " Nov 02 08:00:22.740338 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.740281 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-dshm\") pod \"7db63785-0b2b-4516-b06d-3df792f2019f\" (UID: \"7db63785-0b2b-4516-b06d-3df792f2019f\") " Nov 02 08:00:22.740498 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.740357 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-tmp-dir" pod "7db63785-0b2b-4516-b06d-3df792f2019f" (UID: "7db63785-0b2b-4516-b06d-3df792f2019f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:22.740498 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.740470 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-home" pod "7db63785-0b2b-4516-b06d-3df792f2019f" (UID: "7db63785-0b2b-4516-b06d-3df792f2019f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:22.740601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.740532 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-model-cache" pod "7db63785-0b2b-4516-b06d-3df792f2019f" (UID: "7db63785-0b2b-4516-b06d-3df792f2019f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:22.740816 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.740793 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:22.740900 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.740817 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:22.740900 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.740827 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:22.741935 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.741907 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:22.743109 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.743082 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db63785-0b2b-4516-b06d-3df792f2019f-kube-api-access-ljl5t" pod "7db63785-0b2b-4516-b06d-3df792f2019f" (UID: "7db63785-0b2b-4516-b06d-3df792f2019f"). InnerVolumeSpecName "kube-api-access-ljl5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:00:22.743259 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.743118 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-dshm" pod "7db63785-0b2b-4516-b06d-3df792f2019f" (UID: "7db63785-0b2b-4516-b06d-3df792f2019f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:22.743389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.743362 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db63785-0b2b-4516-b06d-3df792f2019f-tls-certs" pod "7db63785-0b2b-4516-b06d-3df792f2019f" (UID: "7db63785-0b2b-4516-b06d-3df792f2019f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:00:22.842567 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.842526 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db63785-0b2b-4516-b06d-3df792f2019f-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:22.842567 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.842565 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db63785-0b2b-4516-b06d-3df792f2019f-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:22.842779 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:22.842580 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljl5t\" (UniqueName: \"kubernetes.io/projected/7db63785-0b2b-4516-b06d-3df792f2019f-kube-api-access-ljl5t\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:23.366008 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.360680 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:23.548383 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.548346 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:23.592054 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.592024 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" Nov 02 08:00:23.592483 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.592052 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng" event={"ID":"7db63785-0b2b-4516-b06d-3df792f2019f","Type":"ContainerDied","Data":"a890835d3904df1b72ff52717be51cf871100a23bc0cd5a642d67f8b544e3f56"} Nov 02 08:00:23.592483 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.592113 5899 scope.go:122] "RemoveContainer" containerID="37e107ab874b2155c459cba959f1e2c29739c3065b0fc7f0189b6262a3e0430f" Nov 02 08:00:23.631813 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.631735 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb"] Nov 02 08:00:23.632562 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632539 5899 cpu_manager.go:414] "containerMap: removing container" podUID="6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" containerName="istio-proxy" Nov 02 08:00:23.632618 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632566 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" containerName="istio-proxy" Nov 02 08:00:23.632618 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632585 5899 cpu_manager.go:414] "containerMap: removing container" podUID="7db63785-0b2b-4516-b06d-3df792f2019f" containerName="main" Nov 02 08:00:23.632618 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632593 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db63785-0b2b-4516-b06d-3df792f2019f" containerName="main" Nov 02 08:00:23.632618 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632609 5899 cpu_manager.go:414] "containerMap: removing container" podUID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" containerName="cleanup" Nov 02 08:00:23.632737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632617 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" containerName="cleanup" Nov 02 08:00:23.632737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632639 5899 cpu_manager.go:414] "containerMap: removing container" podUID="be1305b8-1ae0-44f4-a860-b19b01e5449e" containerName="cleanup" Nov 02 08:00:23.632737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632646 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1305b8-1ae0-44f4-a860-b19b01e5449e" containerName="cleanup" Nov 02 08:00:23.632737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632660 5899 cpu_manager.go:414] "containerMap: removing container" podUID="df9cec19-431f-4e04-b075-25c4d3759259" containerName="main" Nov 02 08:00:23.632737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632667 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9cec19-431f-4e04-b075-25c4d3759259" containerName="main" Nov 02 08:00:23.632737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632694 5899 cpu_manager.go:414] "containerMap: removing container" podUID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" containerName="cleanup" Nov 02 08:00:23.632737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632702 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" containerName="cleanup" Nov 02 08:00:23.632737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632716 5899 cpu_manager.go:414] "containerMap: removing container" podUID="c36051c8-200c-49f8-b02b-652bc857cedc" containerName="istio-proxy" Nov 02 08:00:23.632737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632723 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c36051c8-200c-49f8-b02b-652bc857cedc" containerName="istio-proxy" Nov 02 08:00:23.632989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632740 5899 cpu_manager.go:414] "containerMap: removing container" podUID="be1305b8-1ae0-44f4-a860-b19b01e5449e" containerName="cleanup" Nov 02 08:00:23.632989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632747 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1305b8-1ae0-44f4-a860-b19b01e5449e" containerName="cleanup" Nov 02 08:00:23.632989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632775 5899 cpu_manager.go:414] "containerMap: removing container" podUID="be1305b8-1ae0-44f4-a860-b19b01e5449e" containerName="cleanup" Nov 02 08:00:23.632989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632782 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1305b8-1ae0-44f4-a860-b19b01e5449e" containerName="cleanup" Nov 02 08:00:23.632989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632798 5899 cpu_manager.go:414] "containerMap: removing container" podUID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" containerName="cleanup" Nov 02 08:00:23.632989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632805 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" containerName="cleanup" Nov 02 08:00:23.632989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632941 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1305b8-1ae0-44f4-a860-b19b01e5449e" containerName="cleanup" Nov 02 08:00:23.632989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632954 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" containerName="cleanup" Nov 02 08:00:23.632989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632965 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f72f7a8-6385-4190-9a7c-beecdd3ef7ef" containerName="istio-proxy" Nov 02 08:00:23.632989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632973 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db63785-0b2b-4516-b06d-3df792f2019f" containerName="main" Nov 02 08:00:23.632989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632984 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" containerName="cleanup" Nov 02 08:00:23.633323 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.632994 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1305b8-1ae0-44f4-a860-b19b01e5449e" containerName="cleanup" Nov 02 08:00:23.633323 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.633003 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9cec19-431f-4e04-b075-25c4d3759259" containerName="main" Nov 02 08:00:23.633323 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.633014 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="c36051c8-200c-49f8-b02b-652bc857cedc" containerName="istio-proxy" Nov 02 08:00:23.638222 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.638155 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" Nov 02 08:00:23.666870 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.666842 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-736f7c\"" Nov 02 08:00:23.671692 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.671656 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb"] Nov 02 08:00:23.678411 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.678376 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng"] Nov 02 08:00:23.682529 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.682492 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-32ea65/auth-test-model-32ea65-kserve-944bd89c-gpbng"] Nov 02 08:00:23.748183 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.747480 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-736f7c-dockercfg-559zh\"" Nov 02 08:00:23.775184 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.771899 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654cd8f0-8663-470c-bcc2-26a96862186d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb\" (UID: \"654cd8f0-8663-470c-bcc2-26a96862186d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" Nov 02 08:00:23.775184 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.772139 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82gf\" (UniqueName: \"kubernetes.io/projected/654cd8f0-8663-470c-bcc2-26a96862186d-kube-api-access-g82gf\") pod \"payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb\" (UID: \"654cd8f0-8663-470c-bcc2-26a96862186d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" Nov 02 08:00:23.788186 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.783683 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:23.824418 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.824379 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db63785-0b2b-4516-b06d-3df792f2019f" path="/var/lib/kubelet/pods/7db63785-0b2b-4516-b06d-3df792f2019f/volumes" Nov 02 08:00:23.825052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.825025 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9cec19-431f-4e04-b075-25c4d3759259" path="/var/lib/kubelet/pods/df9cec19-431f-4e04-b075-25c4d3759259/volumes" Nov 02 08:00:23.873127 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.873093 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82gf\" (UniqueName: \"kubernetes.io/projected/654cd8f0-8663-470c-bcc2-26a96862186d-kube-api-access-g82gf\") pod \"payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb\" (UID: \"654cd8f0-8663-470c-bcc2-26a96862186d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" Nov 02 08:00:23.873403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.873155 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654cd8f0-8663-470c-bcc2-26a96862186d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb\" (UID: \"654cd8f0-8663-470c-bcc2-26a96862186d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" Nov 02 08:00:23.873853 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.873833 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654cd8f0-8663-470c-bcc2-26a96862186d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb\" (UID: \"654cd8f0-8663-470c-bcc2-26a96862186d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" Nov 02 08:00:23.897097 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.897020 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82gf\" (UniqueName: \"kubernetes.io/projected/654cd8f0-8663-470c-bcc2-26a96862186d-kube-api-access-g82gf\") pod \"payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb\" (UID: \"654cd8f0-8663-470c-bcc2-26a96862186d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" Nov 02 08:00:23.954983 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:23.954946 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" Nov 02 08:00:24.263572 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:24.263530 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb"] Nov 02 08:00:24.267060 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:00:24.267032 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654cd8f0_8663_470c_bcc2_26a96862186d.slice/crio-cc0c659d07a9e42448559a9e09e52adf68a78e9e9ef47fcd36c563c9ba2913ab WatchSource:0}: Error finding container cc0c659d07a9e42448559a9e09e52adf68a78e9e9ef47fcd36c563c9ba2913ab: Status 404 returned error can't find the container with id cc0c659d07a9e42448559a9e09e52adf68a78e9e9ef47fcd36c563c9ba2913ab Nov 02 08:00:24.440553 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:24.440503 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf"] Nov 02 08:00:24.508151 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:24.508112 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:24.604082 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:24.603974 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" event={"ID":"654cd8f0-8663-470c-bcc2-26a96862186d","Type":"ContainerStarted","Data":"704f5c3ab130ce383014d819e85521be9fc28adf0134cfe1823b5841371828e9"} Nov 02 08:00:24.604530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:24.604187 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" Nov 02 08:00:24.604530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:24.604228 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" event={"ID":"654cd8f0-8663-470c-bcc2-26a96862186d","Type":"ContainerStarted","Data":"cc0c659d07a9e42448559a9e09e52adf68a78e9e9ef47fcd36c563c9ba2913ab"} Nov 02 08:00:24.648350 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:24.648296 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" podStartSLOduration=1.6482805109999998 podStartE2EDuration="1.648280511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:24.645786457 +0000 UTC m=+2124.944067279" watchObservedRunningTime="2026-11-02 08:00:24.648280511 +0000 UTC m=+2124.946561336" Nov 02 08:00:25.063656 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:25.063619 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:25.696727 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:25.696687 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:25.925921 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:25.925890 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:26.103494 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:26.103449 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:26.290952 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:26.290918 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:27.866038 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:27.866005 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:28.005664 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.005626 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:28.035353 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.035321 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz"] Nov 02 08:00:28.035944 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.035930 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1305b8-1ae0-44f4-a860-b19b01e5449e" containerName="cleanup" Nov 02 08:00:28.035982 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.035957 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd25a8a-cdc8-415b-b736-0fa59bd3719e" containerName="cleanup" Nov 02 08:00:28.039422 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.039402 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" Nov 02 08:00:28.066151 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.066124 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-bb4907\"" Nov 02 08:00:28.066308 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.066151 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-bb4907-dockercfg-4tbvs\"" Nov 02 08:00:28.069422 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.069398 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz"] Nov 02 08:00:28.141120 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.141013 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8xh6\" (UniqueName: \"kubernetes.io/projected/b660cf56-4367-4815-ae30-acbca8e0ce0b-kube-api-access-w8xh6\") pod \"payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz\" (UID: \"b660cf56-4367-4815-ae30-acbca8e0ce0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" Nov 02 08:00:28.141345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.141186 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b660cf56-4367-4815-ae30-acbca8e0ce0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz\" (UID: \"b660cf56-4367-4815-ae30-acbca8e0ce0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" Nov 02 08:00:28.242865 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.242825 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8xh6\" (UniqueName: \"kubernetes.io/projected/b660cf56-4367-4815-ae30-acbca8e0ce0b-kube-api-access-w8xh6\") pod \"payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz\" (UID: \"b660cf56-4367-4815-ae30-acbca8e0ce0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" Nov 02 08:00:28.243049 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.242963 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b660cf56-4367-4815-ae30-acbca8e0ce0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz\" (UID: \"b660cf56-4367-4815-ae30-acbca8e0ce0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" Nov 02 08:00:28.243694 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.243676 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b660cf56-4367-4815-ae30-acbca8e0ce0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz\" (UID: \"b660cf56-4367-4815-ae30-acbca8e0ce0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" Nov 02 08:00:28.265866 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.265836 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8xh6\" (UniqueName: \"kubernetes.io/projected/b660cf56-4367-4815-ae30-acbca8e0ce0b-kube-api-access-w8xh6\") pod \"payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz\" (UID: \"b660cf56-4367-4815-ae30-acbca8e0ce0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" Nov 02 08:00:28.353019 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.352986 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" Nov 02 08:00:28.658938 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:28.658909 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz"] Nov 02 08:00:28.661521 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:00:28.661493 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb660cf56_4367_4815_ae30_acbca8e0ce0b.slice/crio-e92307d1971b872dc87dbc512b2f8a8a8080c2e3313377cc67f9cce16a329aea WatchSource:0}: Error finding container e92307d1971b872dc87dbc512b2f8a8a8080c2e3313377cc67f9cce16a329aea: Status 404 returned error can't find the container with id e92307d1971b872dc87dbc512b2f8a8a8080c2e3313377cc67f9cce16a329aea Nov 02 08:00:29.422810 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:29.422775 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:29.618466 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:29.618433 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:29.640461 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:29.640422 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" event={"ID":"b660cf56-4367-4815-ae30-acbca8e0ce0b","Type":"ContainerStarted","Data":"3c127c305c2e7345b79252321ace8bc40d53b808a5767738717041dbcba28951"} Nov 02 08:00:29.640619 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:29.640488 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" event={"ID":"b660cf56-4367-4815-ae30-acbca8e0ce0b","Type":"ContainerStarted","Data":"e92307d1971b872dc87dbc512b2f8a8a8080c2e3313377cc67f9cce16a329aea"} Nov 02 08:00:29.640619 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:29.640520 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" Nov 02 08:00:29.700227 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:29.700108 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" podStartSLOduration=2.700094659 podStartE2EDuration="2.700094659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:29.699371292 +0000 UTC m=+2129.997652128" watchObservedRunningTime="2026-11-02 08:00:29.700094659 +0000 UTC m=+2129.998375486" Nov 02 08:00:29.779307 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:29.779275 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:29.964557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:29.964479 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:31.193972 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:31.193936 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:31.342023 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:31.341992 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:32.128545 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:32.128509 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:34.001689 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:34.001653 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:35.616867 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:35.616838 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" Nov 02 08:00:36.470666 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:36.470593 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:37.874369 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:37.874332 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:38.315522 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:38.315487 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:39.151940 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.151906 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:39.461951 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.461854 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" podUID="42da3682-ca1a-468b-a4bb-6fe7f75d4279" containerName="main" containerID="cri-o://e1d5a843aea4c10b4f46173b1b205cb2065b261299bab6c4b18799f3481ada08" gracePeriod=45 Nov 02 08:00:39.635387 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.635351 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:39.709489 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.709460 5899 generic.go:356] "Generic (PLEG): container finished" podID="42da3682-ca1a-468b-a4bb-6fe7f75d4279" containerID="e1d5a843aea4c10b4f46173b1b205cb2065b261299bab6c4b18799f3481ada08" exitCode=0 Nov 02 08:00:39.709654 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.709536 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" event={"ID":"42da3682-ca1a-468b-a4bb-6fe7f75d4279","Type":"ContainerDied","Data":"e1d5a843aea4c10b4f46173b1b205cb2065b261299bab6c4b18799f3481ada08"} Nov 02 08:00:39.709654 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.709584 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" event={"ID":"42da3682-ca1a-468b-a4bb-6fe7f75d4279","Type":"ContainerDied","Data":"0380d95cc81980c922f11a76388fbaacd651c4b1e2516fc8ce0507d5a07389de"} Nov 02 08:00:39.709654 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.709601 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0380d95cc81980c922f11a76388fbaacd651c4b1e2516fc8ce0507d5a07389de" Nov 02 08:00:39.742476 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.742442 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 08:00:39.804644 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.804607 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:39.927515 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.927478 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-home\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-home\") pod \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " Nov 02 08:00:39.927720 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.927539 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-model-cache\") pod \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " Nov 02 08:00:39.927720 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.927584 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tls-certs\" (UniqueName: \"kubernetes.io/secret/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tls-certs\") pod \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " Nov 02 08:00:39.927852 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.927754 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tmp-dir\") pod \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " Nov 02 08:00:39.927852 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.927798 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-model-cache" pod "42da3682-ca1a-468b-a4bb-6fe7f75d4279" (UID: "42da3682-ca1a-468b-a4bb-6fe7f75d4279"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:39.927852 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.927827 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42da3682-ca1a-468b-a4bb-6fe7f75d4279-kube-api-access-rkksm\" (UniqueName: \"kubernetes.io/projected/42da3682-ca1a-468b-a4bb-6fe7f75d4279-kube-api-access-rkksm\") pod \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " Nov 02 08:00:39.927852 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.927806 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-home" pod "42da3682-ca1a-468b-a4bb-6fe7f75d4279" (UID: "42da3682-ca1a-468b-a4bb-6fe7f75d4279"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:39.928062 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.927884 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-dshm\") pod \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\" (UID: \"42da3682-ca1a-468b-a4bb-6fe7f75d4279\") " Nov 02 08:00:39.928062 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.927980 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tmp-dir" pod "42da3682-ca1a-468b-a4bb-6fe7f75d4279" (UID: "42da3682-ca1a-468b-a4bb-6fe7f75d4279"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:39.928428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.928407 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:39.928548 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.928429 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:39.928548 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.928443 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:39.930106 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.930084 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-dshm" pod "42da3682-ca1a-468b-a4bb-6fe7f75d4279" (UID: "42da3682-ca1a-468b-a4bb-6fe7f75d4279"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:00:39.930313 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.930292 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tls-certs" pod "42da3682-ca1a-468b-a4bb-6fe7f75d4279" (UID: "42da3682-ca1a-468b-a4bb-6fe7f75d4279"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:00:39.930374 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:39.930335 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42da3682-ca1a-468b-a4bb-6fe7f75d4279-kube-api-access-rkksm" pod "42da3682-ca1a-468b-a4bb-6fe7f75d4279" (UID: "42da3682-ca1a-468b-a4bb-6fe7f75d4279"). InnerVolumeSpecName "kube-api-access-rkksm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:00:40.029240 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:40.029211 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42da3682-ca1a-468b-a4bb-6fe7f75d4279-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:40.029240 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:40.029239 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkksm\" (UniqueName: \"kubernetes.io/projected/42da3682-ca1a-468b-a4bb-6fe7f75d4279-kube-api-access-rkksm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:40.029446 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:40.029250 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42da3682-ca1a-468b-a4bb-6fe7f75d4279-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:00:40.336887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:40.336787 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:40.650733 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:40.650658 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" Nov 02 08:00:40.719614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:40.719579 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf" Nov 02 08:00:40.785078 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:40.785046 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf"] Nov 02 08:00:40.788979 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:40.788950 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-6929a4/auth-test-model-6929a4-kserve-bd76b87b9-h4qhf"] Nov 02 08:00:41.819428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:41.819396 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42da3682-ca1a-468b-a4bb-6fe7f75d4279" path="/var/lib/kubelet/pods/42da3682-ca1a-468b-a4bb-6fe7f75d4279/volumes" Nov 02 08:00:42.105846 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:42.105767 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:43.142151 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:00:43.142116 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:00:43.468795 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:43.468721 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:43.660984 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:43.660950 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:44.166415 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:44.166379 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:45.184731 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.184692 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:45.333199 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.333142 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:45.339068 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.339043 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l"] Nov 02 08:00:45.339641 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.339625 5899 cpu_manager.go:414] "containerMap: removing container" podUID="42da3682-ca1a-468b-a4bb-6fe7f75d4279" containerName="main" Nov 02 08:00:45.339641 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.339640 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42da3682-ca1a-468b-a4bb-6fe7f75d4279" containerName="main" Nov 02 08:00:45.339786 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.339759 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="42da3682-ca1a-468b-a4bb-6fe7f75d4279" containerName="main" Nov 02 08:00:45.343477 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.343458 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" Nov 02 08:00:45.362783 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.362757 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-643355\"" Nov 02 08:00:45.371721 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.371697 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-643355-dockercfg-2vc9j\"" Nov 02 08:00:45.375828 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.375801 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l"] Nov 02 08:00:45.406592 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.406549 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fwk8\" (UniqueName: \"kubernetes.io/projected/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-kube-api-access-6fwk8\") pod \"payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l\" (UID: \"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" Nov 02 08:00:45.406778 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.406615 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l\" (UID: \"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" Nov 02 08:00:45.456224 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.456124 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg"] Nov 02 08:00:45.461503 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.461478 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg"] Nov 02 08:00:45.461832 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.461765 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" Nov 02 08:00:45.508508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.508464 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fwk8\" (UniqueName: \"kubernetes.io/projected/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-kube-api-access-6fwk8\") pod \"payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l\" (UID: \"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" Nov 02 08:00:45.508705 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.508533 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l\" (UID: \"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" Nov 02 08:00:45.508705 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.508634 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp7qn\" (UniqueName: \"kubernetes.io/projected/0915e267-30ab-478c-a749-0fb605399ee4-kube-api-access-vp7qn\") pod \"payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg\" (UID: \"0915e267-30ab-478c-a749-0fb605399ee4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" Nov 02 08:00:45.508705 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.508692 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0915e267-30ab-478c-a749-0fb605399ee4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg\" (UID: \"0915e267-30ab-478c-a749-0fb605399ee4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" Nov 02 08:00:45.509887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.509681 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l\" (UID: \"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" Nov 02 08:00:45.537150 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.533082 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fwk8\" (UniqueName: \"kubernetes.io/projected/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-kube-api-access-6fwk8\") pod \"payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l\" (UID: \"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" Nov 02 08:00:45.609840 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.609800 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp7qn\" (UniqueName: \"kubernetes.io/projected/0915e267-30ab-478c-a749-0fb605399ee4-kube-api-access-vp7qn\") pod \"payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg\" (UID: \"0915e267-30ab-478c-a749-0fb605399ee4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" Nov 02 08:00:45.610046 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.609858 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0915e267-30ab-478c-a749-0fb605399ee4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg\" (UID: \"0915e267-30ab-478c-a749-0fb605399ee4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" Nov 02 08:00:45.610575 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.610555 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0915e267-30ab-478c-a749-0fb605399ee4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg\" (UID: \"0915e267-30ab-478c-a749-0fb605399ee4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" Nov 02 08:00:45.632962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.632931 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp7qn\" (UniqueName: \"kubernetes.io/projected/0915e267-30ab-478c-a749-0fb605399ee4-kube-api-access-vp7qn\") pod \"payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg\" (UID: \"0915e267-30ab-478c-a749-0fb605399ee4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" Nov 02 08:00:45.658859 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.658829 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" Nov 02 08:00:45.783668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.782436 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" Nov 02 08:00:45.963819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:45.963786 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l"] Nov 02 08:00:45.964913 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:00:45.964851 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bda83a4_b696_45c0_b3a2_9b03e75fe3d6.slice/crio-6285ff1378a45c4966ed6783e392230232284ec7a2efcdf60ac740a96ca12df4 WatchSource:0}: Error finding container 6285ff1378a45c4966ed6783e392230232284ec7a2efcdf60ac740a96ca12df4: Status 404 returned error can't find the container with id 6285ff1378a45c4966ed6783e392230232284ec7a2efcdf60ac740a96ca12df4 Nov 02 08:00:46.092279 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:46.092251 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg"] Nov 02 08:00:46.095059 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:00:46.095012 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0915e267_30ab_478c_a749_0fb605399ee4.slice/crio-095f5c3ca076fca10072fe8cf2fce29c1c862779f03cb7fde2d1b4629da84ac5 WatchSource:0}: Error finding container 095f5c3ca076fca10072fe8cf2fce29c1c862779f03cb7fde2d1b4629da84ac5: Status 404 returned error can't find the container with id 095f5c3ca076fca10072fe8cf2fce29c1c862779f03cb7fde2d1b4629da84ac5 Nov 02 08:00:46.194058 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:46.194017 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:46.760988 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:46.760956 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" event={"ID":"0915e267-30ab-478c-a749-0fb605399ee4","Type":"ContainerStarted","Data":"bbaaaea491b9ea6d401bc3e7cce6b94fb9ac479b3957a1f72e452b5d21f1dae9"} Nov 02 08:00:46.761216 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:46.761009 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" Nov 02 08:00:46.761216 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:46.761034 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" event={"ID":"0915e267-30ab-478c-a749-0fb605399ee4","Type":"ContainerStarted","Data":"095f5c3ca076fca10072fe8cf2fce29c1c862779f03cb7fde2d1b4629da84ac5"} Nov 02 08:00:46.762617 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:46.762592 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" event={"ID":"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6","Type":"ContainerStarted","Data":"e00623cdfbdf61f9ee5c311b7cfe483515db4baa83beba6d794000ce8ec8be7a"} Nov 02 08:00:46.762707 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:46.762632 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" event={"ID":"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6","Type":"ContainerStarted","Data":"6285ff1378a45c4966ed6783e392230232284ec7a2efcdf60ac740a96ca12df4"} Nov 02 08:00:46.762707 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:46.762659 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" Nov 02 08:00:46.802565 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:46.802505 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" podStartSLOduration=1.80248965 podStartE2EDuration="1.80248965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:46.800305993 +0000 UTC m=+2147.098586818" watchObservedRunningTime="2026-11-02 08:00:46.80248965 +0000 UTC m=+2147.100770479" Nov 02 08:00:46.841973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:46.841924 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" podStartSLOduration=1.841909944 podStartE2EDuration="1.841909944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:46.839360725 +0000 UTC m=+2147.137641549" watchObservedRunningTime="2026-11-02 08:00:46.841909944 +0000 UTC m=+2147.140190768" Nov 02 08:00:47.936348 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:47.936309 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:49.722791 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:49.722755 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:50.429653 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:50.429610 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:51.128399 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:51.128367 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:51.496971 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:51.496941 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:51.951417 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:51.951335 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:52.641729 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:52.641687 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:52.956696 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:52.956613 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:53.329216 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.329182 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6"] Nov 02 08:00:53.334874 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.334856 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.354317 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.354287 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-736f7c-serving-cert\"" Nov 02 08:00:53.366701 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.366673 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6"] Nov 02 08:00:53.414098 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.414059 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.414302 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.414187 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-metrics-certs\") pod \"maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.414302 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.414226 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6681e487-e123-40f2-82f7-9eb3eaac860b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.414490 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.414464 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nq5f\" (UniqueName: \"kubernetes.io/projected/6681e487-e123-40f2-82f7-9eb3eaac860b-kube-api-access-6nq5f\") pod \"maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.469800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.469763 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:53.515816 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.515781 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6681e487-e123-40f2-82f7-9eb3eaac860b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.516026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.515935 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nq5f\" (UniqueName: \"kubernetes.io/projected/6681e487-e123-40f2-82f7-9eb3eaac860b-kube-api-access-6nq5f\") pod \"maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.516026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.516001 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.516250 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.516041 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-metrics-certs\") pod \"maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.516640 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.516610 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6681e487-e123-40f2-82f7-9eb3eaac860b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.518695 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.518666 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.518809 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.518674 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-metrics-certs\") pod \"maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.540289 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.540255 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nq5f\" (UniqueName: \"kubernetes.io/projected/6681e487-e123-40f2-82f7-9eb3eaac860b-kube-api-access-6nq5f\") pod \"maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.620582 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.620498 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:53.649687 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.649639 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:53.956194 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:53.956154 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6"] Nov 02 08:00:53.957936 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:00:53.957912 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6681e487_e123_40f2_82f7_9eb3eaac860b.slice/crio-014f0ee491857c4eb7e631543d5a1c6bcb5fdee6eb34c5eacfeefc11d10561d6 WatchSource:0}: Error finding container 014f0ee491857c4eb7e631543d5a1c6bcb5fdee6eb34c5eacfeefc11d10561d6: Status 404 returned error can't find the container with id 014f0ee491857c4eb7e631543d5a1c6bcb5fdee6eb34c5eacfeefc11d10561d6 Nov 02 08:00:54.819361 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:54.819325 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" event={"ID":"6681e487-e123-40f2-82f7-9eb3eaac860b","Type":"ContainerStarted","Data":"bcb8e837f9547bcff109b5ad6fc0d280b5302dc35e2d4b659eeeb3931c74b26d"} Nov 02 08:00:54.819772 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:54.819379 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" event={"ID":"6681e487-e123-40f2-82f7-9eb3eaac860b","Type":"ContainerStarted","Data":"014f0ee491857c4eb7e631543d5a1c6bcb5fdee6eb34c5eacfeefc11d10561d6"} Nov 02 08:00:54.819772 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:54.819405 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:00:54.865433 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:54.865379 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" podStartSLOduration=1.415622827 podStartE2EDuration="1.865364821s" totalImagesPullingTime="449.741994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:54.863977747 +0000 UTC m=+2155.162258572" watchObservedRunningTime="2026-11-02 08:00:54.865364821 +0000 UTC m=+2155.163645646" Nov 02 08:00:55.516581 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:55.516543 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:56.836398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:56.836364 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:56.995901 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:56.995867 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:57.552812 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:57.552755 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:57.772213 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:57.772154 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" Nov 02 08:00:57.772377 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:57.772347 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" Nov 02 08:00:58.496454 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:58.496419 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:58.661444 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:58.661387 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:59.337600 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.337566 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt"] Nov 02 08:00:59.341992 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.341969 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.379666 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.379635 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt"] Nov 02 08:00:59.388853 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.388825 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-bb4907-serving-cert\"" Nov 02 08:00:59.429991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.429959 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-metrics-certs\") pod \"maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.430201 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.430008 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d668935-89fc-4909-a4a8-8d0f476490d0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.430201 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.430123 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsl9s\" (UniqueName: \"kubernetes.io/projected/6d668935-89fc-4909-a4a8-8d0f476490d0-kube-api-access-zsl9s\") pod \"maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.430329 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.430299 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.492477 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.492435 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:59.531100 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.531064 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.531588 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.531196 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-metrics-certs\") pod \"maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.531588 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.531233 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d668935-89fc-4909-a4a8-8d0f476490d0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.531588 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.531265 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsl9s\" (UniqueName: \"kubernetes.io/projected/6d668935-89fc-4909-a4a8-8d0f476490d0-kube-api-access-zsl9s\") pod \"maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.531953 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.531925 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d668935-89fc-4909-a4a8-8d0f476490d0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.533530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.533509 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-metrics-certs\") pod \"maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.533603 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.533537 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.554320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.554293 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsl9s\" (UniqueName: \"kubernetes.io/projected/6d668935-89fc-4909-a4a8-8d0f476490d0-kube-api-access-zsl9s\") pod \"maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.633303 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.633217 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:00:59.656833 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.656802 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:00:59.969704 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:00:59.969677 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt"] Nov 02 08:00:59.971336 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:00:59.971311 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d668935_89fc_4909_a4a8_8d0f476490d0.slice/crio-b4755c386e82075525efbe6bd50de4384abcf6b3b2904c6d6bf2fe223231a919 WatchSource:0}: Error finding container b4755c386e82075525efbe6bd50de4384abcf6b3b2904c6d6bf2fe223231a919: Status 404 returned error can't find the container with id b4755c386e82075525efbe6bd50de4384abcf6b3b2904c6d6bf2fe223231a919 Nov 02 08:01:00.831553 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:00.831525 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:01:00.859140 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:00.859104 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" event={"ID":"6d668935-89fc-4909-a4a8-8d0f476490d0","Type":"ContainerStarted","Data":"e0820b5150c23a7804a82f837f50806e8f9875010e01bbeeb88af3d8b648ebd9"} Nov 02 08:01:00.859323 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:00.859173 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" event={"ID":"6d668935-89fc-4909-a4a8-8d0f476490d0","Type":"ContainerStarted","Data":"b4755c386e82075525efbe6bd50de4384abcf6b3b2904c6d6bf2fe223231a919"} Nov 02 08:01:00.859323 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:00.859202 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:01:00.930205 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:00.928362 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" podStartSLOduration=1.5302899540000001 podStartE2EDuration="1.928347021s" totalImagesPullingTime="398.057067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:01:00.925781739 +0000 UTC m=+2161.224062564" watchObservedRunningTime="2026-11-02 08:01:00.928347021 +0000 UTC m=+2161.226627845" Nov 02 08:01:01.510195 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:01.510128 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:03.260882 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:03.260845 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:04.930265 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:04.930228 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn"] Nov 02 08:01:04.935983 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:04.935959 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:04.962478 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:04.962445 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-7a8bb6-serving-cert\"" Nov 02 08:01:04.965259 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:04.965233 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn"] Nov 02 08:01:05.001381 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.001350 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-maas-api-tls\") pod \"maas-api-e2e-autores-7a8bb6-8645578859-dzxrn\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:05.001578 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.001398 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a244a64-c8db-41b3-b9a1-0756edf35fd9-openshift-service-ca\") pod \"maas-api-e2e-autores-7a8bb6-8645578859-dzxrn\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:05.001578 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.001553 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-metrics-certs\") pod \"maas-api-e2e-autores-7a8bb6-8645578859-dzxrn\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:05.001679 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.001609 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnfv2\" (UniqueName: \"kubernetes.io/projected/0a244a64-c8db-41b3-b9a1-0756edf35fd9-kube-api-access-qnfv2\") pod \"maas-api-e2e-autores-7a8bb6-8645578859-dzxrn\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:05.074568 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.074531 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:05.102731 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.102698 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-maas-api-tls\") pod \"maas-api-e2e-autores-7a8bb6-8645578859-dzxrn\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:05.102913 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.102746 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a244a64-c8db-41b3-b9a1-0756edf35fd9-openshift-service-ca\") pod \"maas-api-e2e-autores-7a8bb6-8645578859-dzxrn\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:05.102913 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.102860 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-metrics-certs\") pod \"maas-api-e2e-autores-7a8bb6-8645578859-dzxrn\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:05.103032 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.102918 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnfv2\" (UniqueName: \"kubernetes.io/projected/0a244a64-c8db-41b3-b9a1-0756edf35fd9-kube-api-access-qnfv2\") pod \"maas-api-e2e-autores-7a8bb6-8645578859-dzxrn\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:05.103479 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.103452 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a244a64-c8db-41b3-b9a1-0756edf35fd9-openshift-service-ca\") pod \"maas-api-e2e-autores-7a8bb6-8645578859-dzxrn\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:05.105084 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.105063 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-maas-api-tls\") pod \"maas-api-e2e-autores-7a8bb6-8645578859-dzxrn\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:05.105223 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.105138 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-metrics-certs\") pod \"maas-api-e2e-autores-7a8bb6-8645578859-dzxrn\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:05.125647 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.125619 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnfv2\" (UniqueName: \"kubernetes.io/projected/0a244a64-c8db-41b3-b9a1-0756edf35fd9-kube-api-access-qnfv2\") pod \"maas-api-e2e-autores-7a8bb6-8645578859-dzxrn\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:05.233971 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.233927 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:05.250468 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.250436 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:05.391525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.391488 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:05.563283 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.563241 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn"] Nov 02 08:01:05.563566 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:01:05.563536 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a244a64_c8db_41b3_b9a1_0756edf35fd9.slice/crio-2b56deff5bd43499067d5e1693351852b0812c1c8dcb6ae006b153da998676b0 WatchSource:0}: Error finding container 2b56deff5bd43499067d5e1693351852b0812c1c8dcb6ae006b153da998676b0: Status 404 returned error can't find the container with id 2b56deff5bd43499067d5e1693351852b0812c1c8dcb6ae006b153da998676b0 Nov 02 08:01:05.902501 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:05.902459 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" event={"ID":"0a244a64-c8db-41b3-b9a1-0756edf35fd9","Type":"ContainerStarted","Data":"2b56deff5bd43499067d5e1693351852b0812c1c8dcb6ae006b153da998676b0"} Nov 02 08:01:06.869701 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:06.869671 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:01:06.910702 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:06.910663 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" event={"ID":"0a244a64-c8db-41b3-b9a1-0756edf35fd9","Type":"ContainerStarted","Data":"6063edfd041e98a3b87cdc9953429568d6773b419ad9b87b7e8ee5918547999b"} Nov 02 08:01:06.910850 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:06.910751 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:06.959556 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:06.959494 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" podStartSLOduration=2.6513344500000002 podStartE2EDuration="2.959475091s" totalImagesPullingTime="308.140641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:01:06.955718665 +0000 UTC m=+2167.253999492" watchObservedRunningTime="2026-11-02 08:01:06.959475091 +0000 UTC m=+2167.257755918" Nov 02 08:01:07.155357 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:07.155270 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:09.106872 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:09.106836 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:10.908818 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:10.908734 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:12.738456 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:12.738369 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:12.921009 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:12.920975 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:01:13.141700 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:01:13.141616 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:01:14.616147 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:14.616112 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:16.564449 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:16.564405 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:18.435752 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:18.435698 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:20.061920 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.061879 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f"] Nov 02 08:01:20.067748 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.067722 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.083980 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.083952 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-643355-serving-cert\"" Nov 02 08:01:20.101713 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.101678 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f"] Nov 02 08:01:20.190019 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.189980 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-metrics-certs\") pod \"maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.190256 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.190235 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-maas-api-tls\") pod \"maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.190326 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.190295 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f9a988-8ac6-4969-8e0e-f7a267376b20-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.190388 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.190368 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhjsg\" (UniqueName: \"kubernetes.io/projected/36f9a988-8ac6-4969-8e0e-f7a267376b20-kube-api-access-mhjsg\") pod \"maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.191121 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.191100 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:20.291897 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.291851 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-maas-api-tls\") pod \"maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.292079 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.291909 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f9a988-8ac6-4969-8e0e-f7a267376b20-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.292079 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.291936 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhjsg\" (UniqueName: \"kubernetes.io/projected/36f9a988-8ac6-4969-8e0e-f7a267376b20-kube-api-access-mhjsg\") pod \"maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.292079 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.292002 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-metrics-certs\") pod \"maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.292643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.292615 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f9a988-8ac6-4969-8e0e-f7a267376b20-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.294424 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.294398 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-maas-api-tls\") pod \"maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.294539 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.294516 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-metrics-certs\") pod \"maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.314876 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.314817 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhjsg\" (UniqueName: \"kubernetes.io/projected/36f9a988-8ac6-4969-8e0e-f7a267376b20-kube-api-access-mhjsg\") pod \"maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.330732 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.330700 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:20.383486 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.383442 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:20.473199 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.471630 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:20.703065 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.703039 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f"] Nov 02 08:01:20.705129 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:01:20.705101 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f9a988_8ac6_4969_8e0e_f7a267376b20.slice/crio-61081da5d937e2900e9125a2e24a290a526fe105c3f8849d37307a5062c0a52b WatchSource:0}: Error finding container 61081da5d937e2900e9125a2e24a290a526fe105c3f8849d37307a5062c0a52b: Status 404 returned error can't find the container with id 61081da5d937e2900e9125a2e24a290a526fe105c3f8849d37307a5062c0a52b Nov 02 08:01:20.996490 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:20.996449 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" event={"ID":"36f9a988-8ac6-4969-8e0e-f7a267376b20","Type":"ContainerStarted","Data":"61081da5d937e2900e9125a2e24a290a526fe105c3f8849d37307a5062c0a52b"} Nov 02 08:01:22.003923 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:22.003888 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" event={"ID":"36f9a988-8ac6-4969-8e0e-f7a267376b20","Type":"ContainerStarted","Data":"f9565de779d20ca015ac863d74447b97c3de64c2888db5ad983b41b81c719604"} Nov 02 08:01:22.004331 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:22.003979 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:22.047907 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:22.047839 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" podStartSLOduration=1.737424893 podStartE2EDuration="2.04782053s" totalImagesPullingTime="310.395637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:01:22.045906632 +0000 UTC m=+2182.344187458" watchObservedRunningTime="2026-11-02 08:01:22.04782053 +0000 UTC m=+2182.346101355" Nov 02 08:01:22.153669 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:22.153633 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:23.906347 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:23.906308 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:25.584348 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:25.584309 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:27.257782 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:27.257730 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:28.017095 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:28.017037 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:01:29.131649 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:29.131618 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:31.028065 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:31.028032 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:32.969279 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:32.969246 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq"] Nov 02 08:01:32.974302 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:32.974283 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:32.994271 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:32.994212 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ea9556-serving-cert\"" Nov 02 08:01:33.006908 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.006879 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq"] Nov 02 08:01:33.072049 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.072004 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:33.072261 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.072057 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-metrics-certs\") pod \"maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:33.072261 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.072196 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jrbk\" (UniqueName: \"kubernetes.io/projected/31d07d4a-0d71-4586-95c2-4b67389ec4bd-kube-api-access-8jrbk\") pod \"maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:33.072402 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.072385 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d07d4a-0d71-4586-95c2-4b67389ec4bd-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:33.127579 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.127546 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:33.174058 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.174017 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-metrics-certs\") pod \"maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:33.174311 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.174112 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jrbk\" (UniqueName: \"kubernetes.io/projected/31d07d4a-0d71-4586-95c2-4b67389ec4bd-kube-api-access-8jrbk\") pod \"maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:33.174398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.174349 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d07d4a-0d71-4586-95c2-4b67389ec4bd-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:33.174463 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.174441 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:33.174593 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:01:33.174576 5899 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-serving-cert: secret "maas-api-e2e-extmod-a-ea9556-serving-cert" not found Nov 02 08:01:33.174665 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:01:33.174639 5899 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-maas-api-tls podName:31d07d4a-0d71-4586-95c2-4b67389ec4bd nodeName:}" failed. No retries permitted until 2026-11-02 08:01:33.674620082 +0000 UTC m=+2193.972900893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-maas-api-tls") pod "maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" (UID: "31d07d4a-0d71-4586-95c2-4b67389ec4bd") : secret "maas-api-e2e-extmod-a-ea9556-serving-cert" not found Nov 02 08:01:33.175204 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.175155 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d07d4a-0d71-4586-95c2-4b67389ec4bd-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:33.177026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.176999 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-metrics-certs\") pod \"maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:33.213350 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.213318 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jrbk\" (UniqueName: \"kubernetes.io/projected/31d07d4a-0d71-4586-95c2-4b67389ec4bd-kube-api-access-8jrbk\") pod \"maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:33.367512 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.367477 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:33.505547 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.505509 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:33.681986 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.681897 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:33.684301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.684281 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:33.889116 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:33.889076 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:34.192449 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:34.192391 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq"] Nov 02 08:01:34.194466 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:01:34.194443 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d07d4a_0d71_4586_95c2_4b67389ec4bd.slice/crio-31275362db5c85fc465e862a93ed7e44a90e17b00195d3a3148565102dd93c9b WatchSource:0}: Error finding container 31275362db5c85fc465e862a93ed7e44a90e17b00195d3a3148565102dd93c9b: Status 404 returned error can't find the container with id 31275362db5c85fc465e862a93ed7e44a90e17b00195d3a3148565102dd93c9b Nov 02 08:01:35.101372 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:35.101325 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" event={"ID":"31d07d4a-0d71-4586-95c2-4b67389ec4bd","Type":"ContainerStarted","Data":"83c72e61ee62f6bf096b66c3a202f5e6c27c94741934cdff64c800fe2bfe9461"} Nov 02 08:01:35.101541 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:35.101404 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" event={"ID":"31d07d4a-0d71-4586-95c2-4b67389ec4bd","Type":"ContainerStarted","Data":"31275362db5c85fc465e862a93ed7e44a90e17b00195d3a3148565102dd93c9b"} Nov 02 08:01:35.101541 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:35.101437 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:35.146402 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:35.146346 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" podStartSLOduration=2.8686237439999998 podStartE2EDuration="3.146333517s" totalImagesPullingTime="277.709773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:01:35.143746472 +0000 UTC m=+2195.442027297" watchObservedRunningTime="2026-11-02 08:01:35.146333517 +0000 UTC m=+2195.444614342" Nov 02 08:01:35.309755 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:35.309719 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:37.002874 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:37.002841 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:38.037455 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:38.037412 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:38.768305 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:38.768272 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:39.379560 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:39.379484 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:39.992776 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:39.992742 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:40.334839 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:40.334735 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:40.766549 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:40.766503 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:41.113210 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:41.113117 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:01:42.429571 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:42.429536 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:42.944312 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:42.944278 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:43.141329 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:01:43.141292 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:01:43.529709 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:43.529667 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:43.673223 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:43.673184 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:43.818758 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:43.818666 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:44.082421 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:44.082333 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:44.243857 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:44.243810 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:44.392925 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:44.392811 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:44.904178 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:44.904131 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:45.041735 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:45.041704 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:45.504498 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:45.504462 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:45.641269 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:45.641232 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:46.471345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:46.471312 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:47.469287 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:47.469223 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:47.623743 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:47.623709 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:47.794431 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:47.794391 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:49.599535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:49.599500 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:51.065534 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:51.065499 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:51.536317 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:51.536262 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:52.580006 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.579969 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq"] Nov 02 08:01:52.586658 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.586631 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.604569 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.604536 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq"] Nov 02 08:01:52.613311 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.612979 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7a8bb6\"/\"kube-root-ca.crt\"" Nov 02 08:01:52.613311 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.612980 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7a8bb6\"/\"openshift-service-ca.crt\"" Nov 02 08:01:52.613311 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.613038 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7a8bb6\"/\"default-dockercfg-lmr8v\"" Nov 02 08:01:52.613311 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.613121 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7a8bb6\"/\"autores-model-7a8bb6-kserve-self-signed-certs\"" Nov 02 08:01:52.626548 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.626512 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-dshm\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.626685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.626559 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tmp-dir\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.626685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.626609 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-home\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.626685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.626667 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tls-certs\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.626803 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.626729 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-model-cache\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.626917 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.626898 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4ts\" (UniqueName: \"kubernetes.io/projected/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-kube-api-access-mm4ts\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.664175 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.664132 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:52.728789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.728748 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-model-cache\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.728998 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.728930 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4ts\" (UniqueName: \"kubernetes.io/projected/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-kube-api-access-mm4ts\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.729060 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.729021 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-dshm\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.729060 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.729047 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tmp-dir\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.729126 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.729093 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-home\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.729187 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.729148 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tls-certs\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.729235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.729214 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-model-cache\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.729526 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.729505 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tmp-dir\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.729781 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.729754 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-home\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.732138 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.732114 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-dshm\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.732530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.732510 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tls-certs\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.752084 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.752054 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4ts\" (UniqueName: \"kubernetes.io/projected/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-kube-api-access-mm4ts\") pod \"autores-model-7a8bb6-kserve-6d755b45b5-stgtq\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:52.905478 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:52.900965 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:53.223825 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:53.223797 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq"] Nov 02 08:01:53.225654 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:01:53.225621 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa629e39_d7b6_403e_87fe_7e04eb5fbd7f.slice/crio-5ccddbec98cc7f69ffc37dac5973b82dd6c4463775659ff4ba2c5fa78eb38401 WatchSource:0}: Error finding container 5ccddbec98cc7f69ffc37dac5973b82dd6c4463775659ff4ba2c5fa78eb38401: Status 404 returned error can't find the container with id 5ccddbec98cc7f69ffc37dac5973b82dd6c4463775659ff4ba2c5fa78eb38401 Nov 02 08:01:53.539270 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:53.539238 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:54.239279 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:54.239240 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" event={"ID":"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f","Type":"ContainerStarted","Data":"e3f01dd583df49b1641777a8acd78b50819b1b8db6ba80153a1e4d4413d37bb8"} Nov 02 08:01:54.239279 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:54.239287 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" event={"ID":"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f","Type":"ContainerStarted","Data":"5ccddbec98cc7f69ffc37dac5973b82dd6c4463775659ff4ba2c5fa78eb38401"} Nov 02 08:01:54.280507 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:54.280440 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" podStartSLOduration=2.280425887 podStartE2EDuration="2.280425887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:01:54.279363206 +0000 UTC m=+2214.577644033" watchObservedRunningTime="2026-11-02 08:01:54.280425887 +0000 UTC m=+2214.578706712" Nov 02 08:01:54.901865 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:54.901828 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:01:55.295949 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:55.295915 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:57.088568 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:57.088527 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:01:58.821562 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:01:58.821531 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:00.484998 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:00.484964 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:02.289366 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:02.289330 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:02.902427 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:02.902381 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:02:02.915964 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:02.915932 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:02:03.306676 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:03.306643 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:02:04.622296 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:04.622265 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:05.102433 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:05.102403 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:05.250508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:05.250473 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:06.216329 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:06.216289 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:06.372670 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:06.372636 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:06.530903 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:06.530868 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:06.680443 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:06.680407 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:06.835523 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:06.835432 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:06.994839 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:06.994806 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:07.610216 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:07.610179 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:07.760402 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:07.760361 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:08.828651 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:08.828618 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:09.160515 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:09.160431 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:10.703508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:10.703474 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:12.399915 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:12.399875 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:13.143176 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:02:13.143127 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:02:14.227844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:14.227813 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:15.950985 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:15.950956 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:17.749139 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:17.749101 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:18.404222 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:18.404190 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:18.978543 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:18.978502 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq"] Nov 02 08:02:19.045528 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:19.045490 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:19.616370 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:19.616331 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:19.759635 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:19.759604 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:20.196503 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:20.196470 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:20.555920 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:20.555887 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:21.605088 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:21.605054 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:23.469146 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.469103 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4"] Nov 02 08:02:23.478109 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.478078 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.491966 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.491933 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4"] Nov 02 08:02:23.500304 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.500275 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-2dde84-serving-cert\"" Nov 02 08:02:23.564330 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.564295 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.564528 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.564458 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b327faf-1f96-4df8-b747-f2b810fa3ae4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.564595 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.564548 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-metrics-certs\") pod \"maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.564712 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.564691 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6qh\" (UniqueName: \"kubernetes.io/projected/7b327faf-1f96-4df8-b747-f2b810fa3ae4-kube-api-access-9t6qh\") pod \"maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.583000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.582968 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:23.666358 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.666326 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6qh\" (UniqueName: \"kubernetes.io/projected/7b327faf-1f96-4df8-b747-f2b810fa3ae4-kube-api-access-9t6qh\") pod \"maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.666551 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.666374 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.666551 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.666478 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b327faf-1f96-4df8-b747-f2b810fa3ae4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.666666 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.666578 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-metrics-certs\") pod \"maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.667203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.667147 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b327faf-1f96-4df8-b747-f2b810fa3ae4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.669101 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.669073 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-metrics-certs\") pod \"maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.669216 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.669073 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.693305 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.693272 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6qh\" (UniqueName: \"kubernetes.io/projected/7b327faf-1f96-4df8-b747-f2b810fa3ae4-kube-api-access-9t6qh\") pod \"maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.725345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.725231 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:23.804841 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.804800 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:23.890598 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:23.890556 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:24.040556 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:24.040501 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:24.121498 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:24.121464 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4"] Nov 02 08:02:24.121977 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:02:24.121952 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b327faf_1f96_4df8_b747_f2b810fa3ae4.slice/crio-2ba86d730fb2037106a300b8cdc3880ac79fe822a772ae87fd6cc0b3714bf2ee WatchSource:0}: Error finding container 2ba86d730fb2037106a300b8cdc3880ac79fe822a772ae87fd6cc0b3714bf2ee: Status 404 returned error can't find the container with id 2ba86d730fb2037106a300b8cdc3880ac79fe822a772ae87fd6cc0b3714bf2ee Nov 02 08:02:24.195551 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:24.195497 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:24.434149 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:24.434064 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" event={"ID":"7b327faf-1f96-4df8-b747-f2b810fa3ae4","Type":"ContainerStarted","Data":"2ba86d730fb2037106a300b8cdc3880ac79fe822a772ae87fd6cc0b3714bf2ee"} Nov 02 08:02:24.844307 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:24.844254 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:25.442151 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:25.442119 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" event={"ID":"7b327faf-1f96-4df8-b747-f2b810fa3ae4","Type":"ContainerStarted","Data":"c6bfb3e11596ff1a87b039dfb6798feddfeef71fff9e937a4e8a992d7028adfc"} Nov 02 08:02:25.442366 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:25.442321 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:25.487903 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:25.487849 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" podStartSLOduration=1.969195412 podStartE2EDuration="2.487835509s" totalImagesPullingTime="518.640097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:02:25.485772379 +0000 UTC m=+2245.784053205" watchObservedRunningTime="2026-11-02 08:02:25.487835509 +0000 UTC m=+2245.786116333" Nov 02 08:02:25.727583 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:25.727546 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:25.886576 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:25.886534 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:26.027379 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:26.027280 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:27.989525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:27.989469 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:29.707248 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:29.707213 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:31.454679 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:31.454653 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:02:31.522420 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:31.522381 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:33.253867 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:33.253828 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:34.003435 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.003373 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" podUID="fa629e39-d7b6-403e-87fe-7e04eb5fbd7f" containerName="main" containerID="cri-o://e3f01dd583df49b1641777a8acd78b50819b1b8db6ba80153a1e4d4413d37bb8" gracePeriod=45 Nov 02 08:02:34.284075 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.284048 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:02:34.410227 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.410195 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tmp-dir\") pod \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " Nov 02 08:02:34.410403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.410236 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-dshm\") pod \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " Nov 02 08:02:34.410403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.410300 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-kube-api-access-mm4ts\" (UniqueName: \"kubernetes.io/projected/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-kube-api-access-mm4ts\") pod \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " Nov 02 08:02:34.410525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.410439 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-home\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-home\") pod \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " Nov 02 08:02:34.410525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.410466 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tmp-dir" pod "fa629e39-d7b6-403e-87fe-7e04eb5fbd7f" (UID: "fa629e39-d7b6-403e-87fe-7e04eb5fbd7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:02:34.410525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.410495 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tls-certs\") pod \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " Nov 02 08:02:34.410701 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.410540 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-model-cache\") pod \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\" (UID: \"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f\") " Nov 02 08:02:34.410701 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.410651 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-home" pod "fa629e39-d7b6-403e-87fe-7e04eb5fbd7f" (UID: "fa629e39-d7b6-403e-87fe-7e04eb5fbd7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:02:34.410872 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.410845 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-model-cache" pod "fa629e39-d7b6-403e-87fe-7e04eb5fbd7f" (UID: "fa629e39-d7b6-403e-87fe-7e04eb5fbd7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:02:34.411098 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.411079 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:34.411220 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.411098 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:34.411220 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.411113 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:34.412790 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.412767 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-dshm" pod "fa629e39-d7b6-403e-87fe-7e04eb5fbd7f" (UID: "fa629e39-d7b6-403e-87fe-7e04eb5fbd7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:02:34.412887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.412786 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tls-certs" pod "fa629e39-d7b6-403e-87fe-7e04eb5fbd7f" (UID: "fa629e39-d7b6-403e-87fe-7e04eb5fbd7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:02:34.412887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.412838 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-kube-api-access-mm4ts" pod "fa629e39-d7b6-403e-87fe-7e04eb5fbd7f" (UID: "fa629e39-d7b6-403e-87fe-7e04eb5fbd7f"). InnerVolumeSpecName "kube-api-access-mm4ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:02:34.499356 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.499320 5899 generic.go:356] "Generic (PLEG): container finished" podID="fa629e39-d7b6-403e-87fe-7e04eb5fbd7f" containerID="e3f01dd583df49b1641777a8acd78b50819b1b8db6ba80153a1e4d4413d37bb8" exitCode=0 Nov 02 08:02:34.499520 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.499372 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" Nov 02 08:02:34.499520 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.499400 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" event={"ID":"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f","Type":"ContainerDied","Data":"e3f01dd583df49b1641777a8acd78b50819b1b8db6ba80153a1e4d4413d37bb8"} Nov 02 08:02:34.499520 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.499445 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq" event={"ID":"fa629e39-d7b6-403e-87fe-7e04eb5fbd7f","Type":"ContainerDied","Data":"5ccddbec98cc7f69ffc37dac5973b82dd6c4463775659ff4ba2c5fa78eb38401"} Nov 02 08:02:34.499520 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.499462 5899 scope.go:122] "RemoveContainer" containerID="e3f01dd583df49b1641777a8acd78b50819b1b8db6ba80153a1e4d4413d37bb8" Nov 02 08:02:34.510341 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.510318 5899 scope.go:122] "RemoveContainer" containerID="e3f01dd583df49b1641777a8acd78b50819b1b8db6ba80153a1e4d4413d37bb8" Nov 02 08:02:34.510649 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:02:34.510624 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f01dd583df49b1641777a8acd78b50819b1b8db6ba80153a1e4d4413d37bb8\": container with ID starting with e3f01dd583df49b1641777a8acd78b50819b1b8db6ba80153a1e4d4413d37bb8 not found: ID does not exist" containerID="e3f01dd583df49b1641777a8acd78b50819b1b8db6ba80153a1e4d4413d37bb8" Nov 02 08:02:34.510751 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.510658 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f01dd583df49b1641777a8acd78b50819b1b8db6ba80153a1e4d4413d37bb8"} err="failed to get container status \"e3f01dd583df49b1641777a8acd78b50819b1b8db6ba80153a1e4d4413d37bb8\": rpc error: code = NotFound desc = could not find container \"e3f01dd583df49b1641777a8acd78b50819b1b8db6ba80153a1e4d4413d37bb8\": container with ID starting with e3f01dd583df49b1641777a8acd78b50819b1b8db6ba80153a1e4d4413d37bb8 not found: ID does not exist" Nov 02 08:02:34.512499 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.512479 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:34.512589 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.512504 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm4ts\" (UniqueName: \"kubernetes.io/projected/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-kube-api-access-mm4ts\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:34.512589 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.512521 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:34.555934 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.555905 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq"] Nov 02 08:02:34.559316 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.559293 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-7a8bb6/autores-model-7a8bb6-kserve-6d755b45b5-stgtq"] Nov 02 08:02:34.931358 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:34.931280 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:35.821004 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:35.820969 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa629e39-d7b6-403e-87fe-7e04eb5fbd7f" path="/var/lib/kubelet/pods/fa629e39-d7b6-403e-87fe-7e04eb5fbd7f/volumes" Nov 02 08:02:36.722537 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:36.722506 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:38.522041 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:38.522001 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:40.241083 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:40.241045 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:40.590195 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:40.590084 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp"] Nov 02 08:02:40.590403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:40.590369 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" podUID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" containerName="istio-proxy" containerID="cri-o://5ad83fd6d4435c1265fe17dfbde97ba0b88b0b140d336397c7e28a34919257ba" gracePeriod=30 Nov 02 08:02:41.968679 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:41.968642 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:43.142546 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:02:43.142503 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:02:43.721383 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:43.721342 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:45.234360 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.234320 5899 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:02:45.234742 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.234392 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" podUID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:02:45.444438 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.444397 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:45.868022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.867964 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 08:02:45.951133 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.951102 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istiod-ca-cert\") pod \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " Nov 02 08:02:45.951508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.951175 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-envoy\") pod \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " Nov 02 08:02:45.951508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.951306 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-token\" (UniqueName: \"kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-token\") pod \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " Nov 02 08:02:45.951508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.951346 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-certs\") pod \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " Nov 02 08:02:45.951508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.951387 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-podinfo\") pod \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " Nov 02 08:02:45.951508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.951425 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-kube-api-access-l58gd\" (UniqueName: \"kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-kube-api-access-l58gd\") pod \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " Nov 02 08:02:45.951508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.951499 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-data\") pod \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " Nov 02 08:02:45.951870 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.951564 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-socket\") pod \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " Nov 02 08:02:45.951870 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.951568 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istiod-ca-cert" pod "3489f7c1-5a82-4ac7-8a71-7df03cd28051" (UID: "3489f7c1-5a82-4ac7-8a71-7df03cd28051"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:02:45.951870 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.951673 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-credential-socket\") pod \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\" (UID: \"3489f7c1-5a82-4ac7-8a71-7df03cd28051\") " Nov 02 08:02:45.951870 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.951682 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-certs" pod "3489f7c1-5a82-4ac7-8a71-7df03cd28051" (UID: "3489f7c1-5a82-4ac7-8a71-7df03cd28051"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:02:45.952089 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.951882 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-data" pod "3489f7c1-5a82-4ac7-8a71-7df03cd28051" (UID: "3489f7c1-5a82-4ac7-8a71-7df03cd28051"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:02:45.952331 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.952233 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-credential-socket" pod "3489f7c1-5a82-4ac7-8a71-7df03cd28051" (UID: "3489f7c1-5a82-4ac7-8a71-7df03cd28051"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:02:45.952424 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.952365 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-socket" pod "3489f7c1-5a82-4ac7-8a71-7df03cd28051" (UID: "3489f7c1-5a82-4ac7-8a71-7df03cd28051"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:02:45.952704 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.952665 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:45.952907 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.952889 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:45.952999 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.952915 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:45.952999 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.952944 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:45.954802 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.954774 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-podinfo" pod "3489f7c1-5a82-4ac7-8a71-7df03cd28051" (UID: "3489f7c1-5a82-4ac7-8a71-7df03cd28051"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:02:45.954968 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.954950 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-envoy" pod "3489f7c1-5a82-4ac7-8a71-7df03cd28051" (UID: "3489f7c1-5a82-4ac7-8a71-7df03cd28051"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:02:45.955036 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.954978 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-token" pod "3489f7c1-5a82-4ac7-8a71-7df03cd28051" (UID: "3489f7c1-5a82-4ac7-8a71-7df03cd28051"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:02:45.955983 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:45.955957 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-kube-api-access-l58gd" pod "3489f7c1-5a82-4ac7-8a71-7df03cd28051" (UID: "3489f7c1-5a82-4ac7-8a71-7df03cd28051"). InnerVolumeSpecName "kube-api-access-l58gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:02:46.053910 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.053879 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:46.053910 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.053907 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:46.053910 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.053916 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l58gd\" (UniqueName: \"kubernetes.io/projected/3489f7c1-5a82-4ac7-8a71-7df03cd28051-kube-api-access-l58gd\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:46.054155 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.053926 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:46.054155 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.053935 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489f7c1-5a82-4ac7-8a71-7df03cd28051-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:02:46.592480 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.592444 5899 generic.go:356] "Generic (PLEG): container finished" podID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" containerID="5ad83fd6d4435c1265fe17dfbde97ba0b88b0b140d336397c7e28a34919257ba" exitCode=0 Nov 02 08:02:46.592970 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.592501 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" Nov 02 08:02:46.592970 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.592525 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" event={"ID":"3489f7c1-5a82-4ac7-8a71-7df03cd28051","Type":"ContainerDied","Data":"5ad83fd6d4435c1265fe17dfbde97ba0b88b0b140d336397c7e28a34919257ba"} Nov 02 08:02:46.592970 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.592579 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp" event={"ID":"3489f7c1-5a82-4ac7-8a71-7df03cd28051","Type":"ContainerDied","Data":"351039132ec7fb1cd6709020a0e2a3e0ad3a979e545366a464f0dd6ad390e827"} Nov 02 08:02:46.592970 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.592602 5899 scope.go:122] "RemoveContainer" containerID="5ad83fd6d4435c1265fe17dfbde97ba0b88b0b140d336397c7e28a34919257ba" Nov 02 08:02:46.603005 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.602989 5899 scope.go:122] "RemoveContainer" containerID="5ad83fd6d4435c1265fe17dfbde97ba0b88b0b140d336397c7e28a34919257ba" Nov 02 08:02:46.603265 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:02:46.603245 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad83fd6d4435c1265fe17dfbde97ba0b88b0b140d336397c7e28a34919257ba\": container with ID starting with 5ad83fd6d4435c1265fe17dfbde97ba0b88b0b140d336397c7e28a34919257ba not found: ID does not exist" containerID="5ad83fd6d4435c1265fe17dfbde97ba0b88b0b140d336397c7e28a34919257ba" Nov 02 08:02:46.603319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.603277 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad83fd6d4435c1265fe17dfbde97ba0b88b0b140d336397c7e28a34919257ba"} err="failed to get container status \"5ad83fd6d4435c1265fe17dfbde97ba0b88b0b140d336397c7e28a34919257ba\": rpc error: code = NotFound desc = could not find container \"5ad83fd6d4435c1265fe17dfbde97ba0b88b0b140d336397c7e28a34919257ba\": container with ID starting with 5ad83fd6d4435c1265fe17dfbde97ba0b88b0b140d336397c7e28a34919257ba not found: ID does not exist" Nov 02 08:02:46.667478 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.667446 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp"] Nov 02 08:02:46.680460 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:46.680431 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bfnnp"] Nov 02 08:02:47.251999 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:47.251952 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:47.820089 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:47.820055 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" path="/var/lib/kubelet/pods/3489f7c1-5a82-4ac7-8a71-7df03cd28051/volumes" Nov 02 08:02:49.033025 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:49.032991 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:50.810704 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:50.810668 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:52.627963 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:52.627930 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:54.435342 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:54.435310 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:56.335804 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:56.335770 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:02:58.154299 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:02:58.154264 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:00.085663 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:00.085626 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:00.975302 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:00.975223 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:01.117903 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:01.117868 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:02.025134 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:02.025098 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:03.109124 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:03.109086 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:03.999188 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:03.999140 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt"] Nov 02 08:03:03.999819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:03.999799 5899 cpu_manager.go:414] "containerMap: removing container" podUID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" containerName="istio-proxy" Nov 02 08:03:03.999919 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:03.999820 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" containerName="istio-proxy" Nov 02 08:03:03.999919 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:03.999863 5899 cpu_manager.go:414] "containerMap: removing container" podUID="fa629e39-d7b6-403e-87fe-7e04eb5fbd7f" containerName="main" Nov 02 08:03:03.999919 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:03.999871 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa629e39-d7b6-403e-87fe-7e04eb5fbd7f" containerName="main" Nov 02 08:03:04.000074 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.000047 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa629e39-d7b6-403e-87fe-7e04eb5fbd7f" containerName="main" Nov 02 08:03:04.000128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.000074 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="3489f7c1-5a82-4ac7-8a71-7df03cd28051" containerName="istio-proxy" Nov 02 08:03:04.004864 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.004843 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.037150 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.037123 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-71a41c-serving-cert\"" Nov 02 08:03:04.083137 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.083105 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-metrics-certs\") pod \"maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.083310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.083142 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-maas-api-tls\") pod \"maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.083310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.083258 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/027801e5-b984-456c-a118-953eaf824988-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.083310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.083286 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj6vg\" (UniqueName: \"kubernetes.io/projected/027801e5-b984-456c-a118-953eaf824988-kube-api-access-qj6vg\") pod \"maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.130293 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.130249 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt"] Nov 02 08:03:04.174180 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.174054 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:04.184846 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.184820 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-metrics-certs\") pod \"maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.185011 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.184855 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-maas-api-tls\") pod \"maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.185011 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.184918 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/027801e5-b984-456c-a118-953eaf824988-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.185011 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.184959 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj6vg\" (UniqueName: \"kubernetes.io/projected/027801e5-b984-456c-a118-953eaf824988-kube-api-access-qj6vg\") pod \"maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.186362 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.186333 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/027801e5-b984-456c-a118-953eaf824988-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.196655 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.196628 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-metrics-certs\") pod \"maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.200280 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.200249 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-maas-api-tls\") pod \"maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.210396 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.210374 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj6vg\" (UniqueName: \"kubernetes.io/projected/027801e5-b984-456c-a118-953eaf824988-kube-api-access-qj6vg\") pod \"maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.318336 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.318250 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:04.320346 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.320324 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:04.623850 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.623822 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt"] Nov 02 08:03:04.625970 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:03:04.625940 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027801e5_b984_456c_a118_953eaf824988.slice/crio-4d1657f232fee710c8f67977d21d9a7ce4999a7b6ce49aa7b90853d67c9e7aee WatchSource:0}: Error finding container 4d1657f232fee710c8f67977d21d9a7ce4999a7b6ce49aa7b90853d67c9e7aee: Status 404 returned error can't find the container with id 4d1657f232fee710c8f67977d21d9a7ce4999a7b6ce49aa7b90853d67c9e7aee Nov 02 08:03:04.705557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:04.705514 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" event={"ID":"027801e5-b984-456c-a118-953eaf824988","Type":"ContainerStarted","Data":"4d1657f232fee710c8f67977d21d9a7ce4999a7b6ce49aa7b90853d67c9e7aee"} Nov 02 08:03:05.349593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:05.349554 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:05.714309 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:05.714225 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" event={"ID":"027801e5-b984-456c-a118-953eaf824988","Type":"ContainerStarted","Data":"6e0cbcfc819113643143bb74fb93a4daf6cf8ded472e6ad8dfe2297a9934bc67"} Nov 02 08:03:05.714473 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:05.714332 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:05.761040 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:05.760987 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" podStartSLOduration=2.429961716 podStartE2EDuration="2.760971875s" totalImagesPullingTime="331.010159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:03:05.759271521 +0000 UTC m=+2286.057552357" watchObservedRunningTime="2026-11-02 08:03:05.760971875 +0000 UTC m=+2286.059252699" Nov 02 08:03:06.150051 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:06.150013 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:08.449335 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:08.449293 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:10.019904 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.019867 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn"] Nov 02 08:03:10.020317 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.020106 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" podUID="0a244a64-c8db-41b3-b9a1-0756edf35fd9" containerName="maas-api" containerID="cri-o://6063edfd041e98a3b87cdc9953429568d6773b419ad9b87b7e8ee5918547999b" gracePeriod=30 Nov 02 08:03:10.123355 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.123320 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:10.334238 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.334205 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:03:10.469620 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.469589 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a244a64-c8db-41b3-b9a1-0756edf35fd9-kube-api-access-qnfv2\" (UniqueName: \"kubernetes.io/projected/0a244a64-c8db-41b3-b9a1-0756edf35fd9-kube-api-access-qnfv2\") pod \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " Nov 02 08:03:10.469820 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.469649 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-metrics-certs\") pod \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " Nov 02 08:03:10.469820 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.469684 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-maas-api-tls\") pod \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " Nov 02 08:03:10.469820 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.469763 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a244a64-c8db-41b3-b9a1-0756edf35fd9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a244a64-c8db-41b3-b9a1-0756edf35fd9-openshift-service-ca\") pod \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\" (UID: \"0a244a64-c8db-41b3-b9a1-0756edf35fd9\") " Nov 02 08:03:10.470175 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.470133 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a244a64-c8db-41b3-b9a1-0756edf35fd9-openshift-service-ca" pod "0a244a64-c8db-41b3-b9a1-0756edf35fd9" (UID: "0a244a64-c8db-41b3-b9a1-0756edf35fd9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:03:10.470369 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.470349 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a244a64-c8db-41b3-b9a1-0756edf35fd9-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:03:10.471754 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.471731 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-metrics-certs" pod "0a244a64-c8db-41b3-b9a1-0756edf35fd9" (UID: "0a244a64-c8db-41b3-b9a1-0756edf35fd9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:03:10.471841 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.471751 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-maas-api-tls" pod "0a244a64-c8db-41b3-b9a1-0756edf35fd9" (UID: "0a244a64-c8db-41b3-b9a1-0756edf35fd9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:03:10.471883 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.471852 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a244a64-c8db-41b3-b9a1-0756edf35fd9-kube-api-access-qnfv2" pod "0a244a64-c8db-41b3-b9a1-0756edf35fd9" (UID: "0a244a64-c8db-41b3-b9a1-0756edf35fd9"). InnerVolumeSpecName "kube-api-access-qnfv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:03:10.571557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.571476 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnfv2\" (UniqueName: \"kubernetes.io/projected/0a244a64-c8db-41b3-b9a1-0756edf35fd9-kube-api-access-qnfv2\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:03:10.571557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.571508 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:03:10.571557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.571519 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a244a64-c8db-41b3-b9a1-0756edf35fd9-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:03:10.621658 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.621617 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:10.748218 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.748148 5899 generic.go:356] "Generic (PLEG): container finished" podID="0a244a64-c8db-41b3-b9a1-0756edf35fd9" containerID="6063edfd041e98a3b87cdc9953429568d6773b419ad9b87b7e8ee5918547999b" exitCode=0 Nov 02 08:03:10.748218 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.748190 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" event={"ID":"0a244a64-c8db-41b3-b9a1-0756edf35fd9","Type":"ContainerDied","Data":"6063edfd041e98a3b87cdc9953429568d6773b419ad9b87b7e8ee5918547999b"} Nov 02 08:03:10.748218 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.748216 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" Nov 02 08:03:10.748524 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.748249 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn" event={"ID":"0a244a64-c8db-41b3-b9a1-0756edf35fd9","Type":"ContainerDied","Data":"2b56deff5bd43499067d5e1693351852b0812c1c8dcb6ae006b153da998676b0"} Nov 02 08:03:10.748524 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.748275 5899 scope.go:122] "RemoveContainer" containerID="6063edfd041e98a3b87cdc9953429568d6773b419ad9b87b7e8ee5918547999b" Nov 02 08:03:10.760710 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.760689 5899 scope.go:122] "RemoveContainer" containerID="6063edfd041e98a3b87cdc9953429568d6773b419ad9b87b7e8ee5918547999b" Nov 02 08:03:10.760992 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:03:10.760973 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6063edfd041e98a3b87cdc9953429568d6773b419ad9b87b7e8ee5918547999b\": container with ID starting with 6063edfd041e98a3b87cdc9953429568d6773b419ad9b87b7e8ee5918547999b not found: ID does not exist" containerID="6063edfd041e98a3b87cdc9953429568d6773b419ad9b87b7e8ee5918547999b" Nov 02 08:03:10.761041 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.761001 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6063edfd041e98a3b87cdc9953429568d6773b419ad9b87b7e8ee5918547999b"} err="failed to get container status \"6063edfd041e98a3b87cdc9953429568d6773b419ad9b87b7e8ee5918547999b\": rpc error: code = NotFound desc = could not find container \"6063edfd041e98a3b87cdc9953429568d6773b419ad9b87b7e8ee5918547999b\": container with ID starting with 6063edfd041e98a3b87cdc9953429568d6773b419ad9b87b7e8ee5918547999b not found: ID does not exist" Nov 02 08:03:10.819973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.819937 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn"] Nov 02 08:03:10.832872 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.832796 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a8bb6-8645578859-dzxrn"] Nov 02 08:03:10.979105 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:10.979072 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:11.228799 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:11.228767 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:11.726686 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:11.726664 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:03:11.822009 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:11.821978 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a244a64-c8db-41b3-b9a1-0756edf35fd9" path="/var/lib/kubelet/pods/0a244a64-c8db-41b3-b9a1-0756edf35fd9/volumes" Nov 02 08:03:12.208128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:12.208045 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:12.399818 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:12.399786 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:12.606293 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:12.606260 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:12.821564 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:12.821523 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:12.987305 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:12.987267 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:13.143587 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:03:13.143546 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:03:13.152708 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:13.152669 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:13.667493 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:13.667458 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:13.925857 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:13.925778 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:15.296000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:15.295968 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:15.495026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:15.494990 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:17.221264 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:17.221229 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:17.437880 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:17.437711 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:17.621318 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:17.621278 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:17.809147 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:17.809110 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:17.995362 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:17.995328 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:18.174528 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:18.174493 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:18.487263 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:18.487230 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:18.849015 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:18.848912 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:19.136120 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:19.136032 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:19.598819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:19.598787 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:19.973888 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:19.973802 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:20.126966 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:20.126930 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:20.270305 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:20.270267 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:20.675084 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:20.675008 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:20.814004 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:20.813968 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:21.089898 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:21.089860 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:21.234072 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:21.234039 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:21.851393 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:21.851361 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:22.327987 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:22.327947 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:22.476917 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:22.476881 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:23.029220 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:23.029188 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:24.913530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:24.913499 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:26.491287 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:26.491248 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:26.635916 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:26.635885 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:26.824613 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:26.824532 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:28.332667 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:28.332634 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:28.492809 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:28.492773 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:28.740764 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:28.740733 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:30.498945 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:30.498897 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:32.373540 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:32.373495 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:34.040850 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:34.040818 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:35.810754 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:35.810715 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:37.687593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:37.687516 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:39.393362 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:39.393324 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:41.238498 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:41.238458 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:43.141298 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:03:43.141263 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:03:43.215776 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:43.215734 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:45.607175 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:45.607132 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:47.815629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:47.815568 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:48.337668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:48.337627 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:49.378631 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:49.378575 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:50.404705 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:50.404646 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:52.470284 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:52.470241 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:54.354653 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:54.354615 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:55.840692 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:55.840650 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:56.225392 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:56.225322 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:56.991252 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:56.991204 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:57.143609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:57.143578 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:57.287500 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:57.287380 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:57.557806 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:57.557726 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:57.716881 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:57.716852 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:57.900999 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:57.900917 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:58.229294 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:58.229266 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:58.489951 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:58.489869 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:58.642701 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:58.642653 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:59.041875 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:59.041843 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:59.174989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:59.174957 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:03:59.965755 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:03:59.965708 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:00.534084 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:00.534051 5899 scope.go:122] "RemoveContainer" containerID="e1d5a843aea4c10b4f46173b1b205cb2065b261299bab6c4b18799f3481ada08" Nov 02 08:04:00.675638 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:00.675605 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:02.519926 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:02.519887 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:04.225458 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:04.225346 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:05.475189 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:05.475138 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:06.016448 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:06.016412 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:07.829611 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:07.829577 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:09.442842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.442809 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf"] Nov 02 08:04:09.443445 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.443429 5899 cpu_manager.go:414] "containerMap: removing container" podUID="0a244a64-c8db-41b3-b9a1-0756edf35fd9" containerName="maas-api" Nov 02 08:04:09.443493 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.443446 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a244a64-c8db-41b3-b9a1-0756edf35fd9" containerName="maas-api" Nov 02 08:04:09.443586 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.443577 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a244a64-c8db-41b3-b9a1-0756edf35fd9" containerName="maas-api" Nov 02 08:04:09.447426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.447410 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:09.480198 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.477941 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-0f2e06-serving-cert\"" Nov 02 08:04:09.480949 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.480925 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf"] Nov 02 08:04:09.520318 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.520288 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:09.520475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.520325 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:09.520475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.520435 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b057d2fb-2171-4165-8185-070d04931f5f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:09.520475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.520464 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwmwx\" (UniqueName: \"kubernetes.io/projected/b057d2fb-2171-4165-8185-070d04931f5f-kube-api-access-vwmwx\") pod \"maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:09.567807 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.567774 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:09.622266 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.622231 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:09.622266 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.622267 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:09.622525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.622328 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b057d2fb-2171-4165-8185-070d04931f5f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:09.622525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.622356 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwmwx\" (UniqueName: \"kubernetes.io/projected/b057d2fb-2171-4165-8185-070d04931f5f-kube-api-access-vwmwx\") pod \"maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:09.622525 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:04:09.622397 5899 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-serving-cert: secret "maas-api-e2e-extmod-a-0f2e06-serving-cert" not found Nov 02 08:04:09.622525 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:04:09.622481 5899 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-maas-api-tls podName:b057d2fb-2171-4165-8185-070d04931f5f nodeName:}" failed. No retries permitted until 2026-11-02 08:04:10.122458242 +0000 UTC m=+2350.420739060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-maas-api-tls") pod "maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" (UID: "b057d2fb-2171-4165-8185-070d04931f5f") : secret "maas-api-e2e-extmod-a-0f2e06-serving-cert" not found Nov 02 08:04:09.622988 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.622971 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b057d2fb-2171-4165-8185-070d04931f5f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:09.624741 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.624723 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:09.647772 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.647739 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwmwx\" (UniqueName: \"kubernetes.io/projected/b057d2fb-2171-4165-8185-070d04931f5f-kube-api-access-vwmwx\") pod \"maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:09.710254 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.710145 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:09.847686 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:09.847644 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:10.128533 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:10.128490 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:10.130996 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:10.130967 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:10.362186 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:10.360481 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:10.690400 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:10.690287 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf"] Nov 02 08:04:10.700027 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:04:10.699990 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb057d2fb_2171_4165_8185_070d04931f5f.slice/crio-b6dcafcbb80fd94e0a2e4f43d23798592dc5f243f57d5216533977a98c6b4546 WatchSource:0}: Error finding container b6dcafcbb80fd94e0a2e4f43d23798592dc5f243f57d5216533977a98c6b4546: Status 404 returned error can't find the container with id b6dcafcbb80fd94e0a2e4f43d23798592dc5f243f57d5216533977a98c6b4546 Nov 02 08:04:11.141873 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:11.141836 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" event={"ID":"b057d2fb-2171-4165-8185-070d04931f5f","Type":"ContainerStarted","Data":"b6dcafcbb80fd94e0a2e4f43d23798592dc5f243f57d5216533977a98c6b4546"} Nov 02 08:04:11.676737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:11.676635 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:12.148805 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:12.148765 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" event={"ID":"b057d2fb-2171-4165-8185-070d04931f5f","Type":"ContainerStarted","Data":"51cb6aa8c9ec9fb09fa2beb31547a38a86a96afcd44626e3402dbb87100c910e"} Nov 02 08:04:12.149291 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:12.148935 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:12.194385 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:12.194332 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" podStartSLOduration=2.491566032 podStartE2EDuration="3.194318734s" totalImagesPullingTime="702.752702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:04:12.19177439 +0000 UTC m=+2352.490055216" watchObservedRunningTime="2026-11-02 08:04:12.194318734 +0000 UTC m=+2352.492599558" Nov 02 08:04:13.141555 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:04:13.141517 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:04:13.527276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:13.527243 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:15.153645 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:15.153605 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:16.893233 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:16.893197 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:18.162185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:18.162140 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:04:18.557785 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:18.557738 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:18.691936 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:18.691905 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:18.864953 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:18.864862 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:20.507639 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:20.507602 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:22.235650 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:22.235616 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:24.113461 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:24.113427 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:26.019847 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:26.019812 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:27.814836 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:27.814796 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:29.570315 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:29.570260 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:31.316410 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:31.316378 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:33.128821 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:33.128776 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:34.846408 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:34.846371 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:36.615332 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:36.615297 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:38.318180 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:38.318134 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:40.098724 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:40.098680 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:41.853001 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:41.852968 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:43.141461 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:04:43.141427 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:04:43.603737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:43.603687 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:45.289568 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:45.289538 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:47.040839 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:47.040806 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:48.852383 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:48.852352 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:50.704464 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:50.704419 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:52.526321 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:52.526283 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:53.365416 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:53.365376 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:54.618213 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:54.618181 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:54.766576 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:54.766524 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:55.247029 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:55.246988 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:55.442423 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:55.442383 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:55.922538 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:55.922498 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:56.718291 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:56.718246 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:57.295843 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:57.295807 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:58.487728 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:58.487683 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:58.652630 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:58.652590 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:58.916457 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:58.916361 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:04:59.150423 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:04:59.150391 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:01.193686 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:01.193624 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:01.351956 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:01.351923 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:01.803683 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:01.803647 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:01.946196 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:01.946146 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:03.281116 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:03.281077 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:03.447342 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:03.447303 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:03.591984 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:03.591871 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:05.143635 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:05.143603 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:05.598868 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:05.598831 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:06.893764 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:06.893727 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:08.541407 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:08.541374 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:08.671407 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:08.671366 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:08.816836 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:08.816756 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:10.335834 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:10.335795 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:10.471819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:10.471789 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:10.611311 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:10.611238 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:10.751749 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:10.751713 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:12.406867 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:12.406833 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:13.140950 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:05:13.140915 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:05:14.102764 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:14.102730 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:15.845345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:15.845314 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:16.305707 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:16.305666 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:16.451746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:16.451704 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:17.122152 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:17.122114 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:17.292426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:17.292377 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:17.932860 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:17.932826 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:18.138241 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:18.138203 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:19.832933 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:19.832895 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:20.124332 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:20.124246 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:20.268086 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:20.268038 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:22.002536 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:22.002506 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:22.439819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:22.439732 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:22.950765 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:22.950721 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:23.984571 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:23.984534 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:24.208726 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:24.208697 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:24.654000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:24.653952 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:24.860774 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:24.860722 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:25.010117 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.010084 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:25.170853 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.170817 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:25.340873 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.340767 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:25.501151 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.501117 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq"] Nov 02 08:05:25.505607 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.505586 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.526649 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.526621 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-643355-openshift-default-dockercfg-7ch4b\"" Nov 02 08:05:25.542530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.542504 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq"] Nov 02 08:05:25.599072 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.598989 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-credential-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.599072 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.599062 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57768ba3-8718-4280-8205-1827026315b9-istiod-ca-cert\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.599312 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.599218 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-certs\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.599312 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.599240 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-data\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.599312 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.599261 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57768ba3-8718-4280-8205-1827026315b9-istio-podinfo\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.599481 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.599414 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcqgn\" (UniqueName: \"kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-kube-api-access-xcqgn\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.599481 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.599458 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-envoy\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.599573 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.599537 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-istio-token\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.599573 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.599558 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.701300 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.701261 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-certs\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.701503 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.701313 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-data\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.701503 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.701370 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57768ba3-8718-4280-8205-1827026315b9-istio-podinfo\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.701503 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.701449 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcqgn\" (UniqueName: \"kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-kube-api-access-xcqgn\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.701503 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.701486 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-envoy\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.701758 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.701732 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-certs\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.701834 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.701764 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-data\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.701834 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.701812 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-istio-token\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.701963 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.701846 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.701963 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.701891 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-credential-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.702079 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.701978 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57768ba3-8718-4280-8205-1827026315b9-istiod-ca-cert\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.702229 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.702208 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.702409 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.702367 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-credential-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.702704 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.702683 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57768ba3-8718-4280-8205-1827026315b9-istiod-ca-cert\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.704197 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.704145 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57768ba3-8718-4280-8205-1827026315b9-istio-podinfo\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.704303 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.704257 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-envoy\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.728335 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.728306 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-istio-token\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.729619 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.729598 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcqgn\" (UniqueName: \"kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-kube-api-access-xcqgn\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.819494 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.819432 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:25.855876 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:25.855796 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:26.001806 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:26.001771 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:26.158628 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:26.158591 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq"] Nov 02 08:05:26.162101 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:05:26.161397 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57768ba3_8718_4280_8205_1827026315b9.slice/crio-9a3d70786b1f8c96fe49fb02a05960691285f807704d097ba4f8c4d9200f9096 WatchSource:0}: Error finding container 9a3d70786b1f8c96fe49fb02a05960691285f807704d097ba4f8c4d9200f9096: Status 404 returned error can't find the container with id 9a3d70786b1f8c96fe49fb02a05960691285f807704d097ba4f8c4d9200f9096 Nov 02 08:05:26.164948 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:26.164905 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:05:26.165064 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:26.164996 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:05:26.165128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:26.165065 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:05:26.635364 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:26.635324 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" event={"ID":"57768ba3-8718-4280-8205-1827026315b9","Type":"ContainerStarted","Data":"8c9694f65a210fc374a040ee254de633b63ba60cb8360e0d7df37b5a9d5e7b76"} Nov 02 08:05:26.635551 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:26.635391 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" event={"ID":"57768ba3-8718-4280-8205-1827026315b9","Type":"ContainerStarted","Data":"9a3d70786b1f8c96fe49fb02a05960691285f807704d097ba4f8c4d9200f9096"} Nov 02 08:05:26.686262 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:26.686211 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" podStartSLOduration=1.686196775 podStartE2EDuration="1.686196775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:26.683785046 +0000 UTC m=+2426.982065871" watchObservedRunningTime="2026-11-02 08:05:26.686196775 +0000 UTC m=+2426.984477666" Nov 02 08:05:26.822410 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:26.822371 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:27.823774 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:27.823737 5899 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 08:05:27.824244 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:27.823795 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" podUID="57768ba3-8718-4280-8205-1827026315b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 08:05:27.913861 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:27.913805 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:28.480672 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:28.480635 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:28.822644 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:28.822552 5899 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 08:05:28.822644 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:28.822608 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" podUID="57768ba3-8718-4280-8205-1827026315b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 08:05:29.822073 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:29.822041 5899 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 08:05:29.822483 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:29.822092 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" podUID="57768ba3-8718-4280-8205-1827026315b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 08:05:29.824983 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:29.824958 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:30.486399 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:30.486359 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:30.653384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:30.653350 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:30.821958 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:30.821874 5899 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 08:05:30.821958 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:30.821928 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" podUID="57768ba3-8718-4280-8205-1827026315b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 08:05:31.594588 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:31.594549 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:31.822336 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:31.822303 5899 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 08:05:31.822755 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:31.822357 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" podUID="57768ba3-8718-4280-8205-1827026315b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 08:05:32.827308 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:32.827276 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:32.827764 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:32.827742 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:32.828348 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:32.828329 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:05:33.410069 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:33.410030 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:35.181960 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:35.181926 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:36.899078 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:36.899031 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:37.035698 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:37.035670 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:38.629492 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:38.629456 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:40.648609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:40.648574 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:42.245416 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.245379 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl"] Nov 02 08:05:42.249895 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.249871 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.270866 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.270836 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl"] Nov 02 08:05:42.280681 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.280643 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-2365fa-serving-cert\"" Nov 02 08:05:42.348657 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.348624 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc598\" (UniqueName: \"kubernetes.io/projected/760654b4-d894-489c-bf1b-3dd442fbe8da-kube-api-access-zc598\") pod \"maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.348657 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.348661 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760654b4-d894-489c-bf1b-3dd442fbe8da-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.348891 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.348788 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.348956 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.348917 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-metrics-certs\") pod \"maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.374647 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.374588 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:42.449853 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.449816 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760654b4-d894-489c-bf1b-3dd442fbe8da-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.450044 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.449911 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.450044 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.449992 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-metrics-certs\") pod \"maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.450206 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.450123 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc598\" (UniqueName: \"kubernetes.io/projected/760654b4-d894-489c-bf1b-3dd442fbe8da-kube-api-access-zc598\") pod \"maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.450729 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.450688 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760654b4-d894-489c-bf1b-3dd442fbe8da-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.452700 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.452672 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.452860 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.452704 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-metrics-certs\") pod \"maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.476558 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.476521 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc598\" (UniqueName: \"kubernetes.io/projected/760654b4-d894-489c-bf1b-3dd442fbe8da-kube-api-access-zc598\") pod \"maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.511215 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.511109 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:42.563973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.563940 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:42.874495 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.874469 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl"] Nov 02 08:05:42.876591 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:05:42.876560 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760654b4_d894_489c_bf1b_3dd442fbe8da.slice/crio-503758b263877c966d62893234ec0e38d104a22a11578815e2fa5bd36198fbc1 WatchSource:0}: Error finding container 503758b263877c966d62893234ec0e38d104a22a11578815e2fa5bd36198fbc1: Status 404 returned error can't find the container with id 503758b263877c966d62893234ec0e38d104a22a11578815e2fa5bd36198fbc1 Nov 02 08:05:42.878094 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:42.878075 5899 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:05:43.144989 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:05:43.144898 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:05:43.773384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:43.773345 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" event={"ID":"760654b4-d894-489c-bf1b-3dd442fbe8da","Type":"ContainerStarted","Data":"568a4fde7b923ec39355beefba74136998ed30643f3b1d5d61cca5562a1d2481"} Nov 02 08:05:43.773819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:43.773399 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" event={"ID":"760654b4-d894-489c-bf1b-3dd442fbe8da","Type":"ContainerStarted","Data":"503758b263877c966d62893234ec0e38d104a22a11578815e2fa5bd36198fbc1"} Nov 02 08:05:43.773819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:43.773438 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:43.818929 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:43.818879 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" podStartSLOduration=1.3818463109999999 podStartE2EDuration="1.818866976s" totalImagesPullingTime="437.020665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:43.817088657 +0000 UTC m=+2444.115369495" watchObservedRunningTime="2026-11-02 08:05:43.818866976 +0000 UTC m=+2444.117147801" Nov 02 08:05:44.314045 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:44.314007 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:46.123529 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:46.123493 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:47.838808 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:47.838772 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:49.699882 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:49.699834 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:49.785945 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:49.785915 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:05:51.694912 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:51.694870 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:52.084792 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:52.084750 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:52.288018 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:52.287977 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:52.945833 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:52.945795 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:54.265849 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:54.265816 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:56.037103 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:56.037066 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:57.806372 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:57.806339 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:58.517659 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:58.517602 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:05:58.666528 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:05:58.666490 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:00.009970 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:00.009927 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:01.788808 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:01.788770 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:03.555328 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:03.555295 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:05.253789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:05.253747 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:05.407522 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:05.407475 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:05.543081 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:05.543003 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:05.739192 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:05.738875 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:06.018468 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:06.018435 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:07.122401 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:07.122372 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:08.868536 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:08.868504 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:10.908589 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:10.908553 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:12.896662 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:12.896618 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:13.042688 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:13.042653 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:13.141655 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:06:13.141619 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:06:13.204241 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:13.204131 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:13.344375 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:13.344341 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:14.683978 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:14.683936 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:14.838305 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:14.838262 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:15.232244 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:15.232199 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:17.012809 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:17.012776 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:18.766799 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:18.766766 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:19.387799 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:19.387768 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:19.577592 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:19.577560 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:19.798734 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:19.798694 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:19.969622 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:19.969584 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:21.018625 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:21.018587 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:21.158537 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:21.158505 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:22.952274 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:22.952240 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:24.866542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:24.866509 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:25.016389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:25.016353 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:25.456981 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:25.456947 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:26.699450 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:26.699416 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:26.835133 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:26.835098 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:28.687485 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:28.687447 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:31.032909 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:31.032873 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:32.111717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:32.111679 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:32.272103 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:32.272067 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:32.415338 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:32.415265 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:32.970533 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:32.970488 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:34.817370 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:34.817336 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:35.472614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:35.472581 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:36.616510 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:36.616478 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:36.763317 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:36.763273 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:36.924076 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:36.923969 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:37.184334 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:37.184251 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:37.373356 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:37.373317 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:37.516450 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:37.516417 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:38.032026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:38.031990 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:38.181309 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:38.181253 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:38.537279 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:38.537243 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:38.670001 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:38.669949 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:38.819198 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:38.819088 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:38.969541 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:38.969488 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:39.130977 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:39.130863 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:39.280958 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:39.280925 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:39.724522 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:39.724483 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:40.128542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:40.128493 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:40.288779 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:40.288733 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:40.440403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:40.440321 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:40.582811 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:40.582762 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:40.772151 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:40.772107 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:40.917480 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:40.917451 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:41.064399 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:41.064303 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:41.618536 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:41.618500 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:41.852199 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:41.851386 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:42.829146 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:42.829099 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:43.136478 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:43.136400 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:43.142018 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:06:43.141987 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:06:43.324652 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:43.324616 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:44.128810 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:44.128775 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:44.316538 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:44.316500 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:44.459357 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:44.459266 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:44.628576 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:44.628535 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:45.060646 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:45.060615 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:46.515491 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:46.515458 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:46.653429 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:46.653397 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:46.784114 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:46.784027 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:46.920375 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:46.920331 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:48.576221 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:48.576185 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:48.712727 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:48.712696 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:50.475373 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:50.475335 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:50.762780 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:50.762747 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:51.471104 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:51.471069 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:51.613398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:51.613362 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:52.116629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:52.116590 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:53.037714 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:53.037672 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:53.911154 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:53.911122 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:54.054153 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:54.054120 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:54.414773 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:54.414740 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:54.601660 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:54.601623 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:55.201049 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:55.200706 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:55.604647 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:55.604612 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:57.217856 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:57.217821 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:58.044376 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:58.044339 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:58.205648 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:58.205598 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:59.123327 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:59.123295 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:59.287276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:59.287245 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:06:59.481548 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:06:59.481521 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:00.878187 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:00.878128 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:02.815238 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:02.815200 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:04.667467 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:04.667434 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:05.867121 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:05.867083 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:06.221885 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:06.221802 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:06.450061 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:06.450030 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:06.637258 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:06.637221 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:06.896366 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:06.896263 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:07.188731 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:07.188657 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:07.391302 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:07.391269 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:07.957782 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:07.957747 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:08.128667 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:08.128631 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:09.324193 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:09.324123 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:09.500610 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:09.500573 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:09.965551 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:09.965517 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:11.863450 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:11.863369 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:13.142922 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:07:13.142879 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:07:13.753413 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:13.753363 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:15.558132 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:15.558098 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:17.551935 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:17.551894 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:19.377035 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:19.376997 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:21.330518 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:21.330478 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:23.213073 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:23.213034 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:25.258535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:25.258500 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:27.030399 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:27.030363 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:28.769104 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:28.769048 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:30.478887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:30.478854 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:32.223820 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:32.223787 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:33.913052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:33.913004 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:34.235700 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:34.235671 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:34.379070 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:34.379031 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:35.695413 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:35.695373 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:37.520733 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:37.520689 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:39.277996 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:39.277960 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:41.014465 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:41.014433 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:42.962783 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:42.962750 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:43.141333 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:07:43.141296 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:07:44.672358 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:44.672320 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:46.168412 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:46.168378 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:46.312130 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:46.312097 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:46.613617 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:46.613575 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:46.776173 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:46.776139 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:48.518057 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:48.518004 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:50.325718 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:50.325683 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:52.416298 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:52.416241 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:53.999431 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:53.999397 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:54.218187 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:54.218127 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:54.366364 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:54.366284 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:56.174013 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:56.173981 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:58.020124 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:58.020094 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:07:59.908209 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:07:59.908147 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:00.582076 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:00.582021 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:00.799741 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:00.799693 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:01.008317 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:01.008270 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:01.207773 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:01.207743 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:01.534600 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:01.534570 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:01.990787 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:01.990745 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:02.265514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:02.265393 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:02.593243 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:02.593061 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:03.947441 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:03.947393 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:04.365708 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:04.365672 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:06.031697 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:06.031662 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:06.363857 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:06.363786 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:06.510345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:06.510317 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:06.750435 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:06.750393 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:06.891578 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:06.891519 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:08.415035 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:08.414975 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:10.319924 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:10.319889 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:10.462910 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:10.462878 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:12.061795 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:12.061758 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:13.141485 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:08:13.141452 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:08:13.850887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:13.850852 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:15.523523 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:15.523492 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:17.229447 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:17.229412 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:17.382173 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:17.382125 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:17.520117 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:17.520026 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:17.709318 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:17.709283 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:18.100112 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:18.100075 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:18.244373 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:18.244341 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:18.385330 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:18.385244 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:18.548373 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:18.548334 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:18.795402 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:18.795365 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:19.200662 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:19.200576 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:19.450564 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:19.450529 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:19.589634 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:19.589596 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:20.992550 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:20.992520 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:21.519297 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:21.519266 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:23.255560 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:23.255527 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:25.142569 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:25.142540 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:26.904098 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:26.904069 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:27.658830 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:27.658795 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:27.807967 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:27.807926 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:27.959376 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:27.959297 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:28.345590 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:28.345555 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:28.507281 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:28.507247 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:28.675403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:28.675322 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:28.902830 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:28.902801 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:29.058094 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:29.058062 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:29.218973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:29.218942 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:29.691393 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:29.691349 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:29.820381 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:29.820350 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:30.996670 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:30.996631 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:31.139706 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:31.139674 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:31.278327 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:31.278226 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:32.964547 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:32.964517 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:34.800237 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:34.800202 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:36.815988 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:36.815951 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:38.595429 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:38.595370 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:40.558941 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:40.558886 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:42.358386 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:42.358347 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:43.141194 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:08:43.141135 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:08:44.104691 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:44.104658 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:45.792816 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:45.792781 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:47.588023 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:47.587969 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:49.329729 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:49.329686 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:51.071645 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:51.071609 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:52.907515 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:52.907484 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:54.718415 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:54.718380 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:56.563469 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:56.563436 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:57.562127 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:57.562093 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:57.702563 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:57.702530 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:08:58.698868 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:08:58.698831 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:00.369978 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:00.369942 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:02.119270 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:02.119210 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:03.823971 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:03.823936 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:05.597314 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:05.597279 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:06.224666 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:06.224634 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:06.490640 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:06.490605 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:07.607172 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:07.607137 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:09.351927 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:09.351887 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:09.585721 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:09.585673 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:09.784429 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:09.784391 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:09.935522 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:09.935481 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:10.813328 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:10.813291 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:10.971310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:10.971278 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:11.143847 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:11.143765 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:11.431624 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:11.431531 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:11.667518 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:11.667484 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:13.141422 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:09:13.141386 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:09:13.409904 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:13.409822 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:15.192094 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:15.192054 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:17.201508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:17.201471 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:17.926584 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:17.926552 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:18.055507 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:18.055473 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:18.198425 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:18.198338 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:19.008086 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:19.008023 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:20.809745 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:20.809711 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:22.557513 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:22.557481 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:24.280671 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:24.280624 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:24.425208 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:24.425173 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:24.561567 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:24.561493 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:25.321920 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:25.321882 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:25.463996 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:25.463959 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:26.379491 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:26.379455 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:26.761228 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:26.761191 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:28.219068 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:28.219029 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:30.089713 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:30.089679 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:31.566861 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:31.566823 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:31.836510 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:31.836426 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:32.039471 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:32.039435 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:32.809931 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:32.809898 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:32.980472 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:32.980436 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:33.441207 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:33.441174 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:34.256498 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:34.256463 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:36.111955 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:36.111919 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:37.933272 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:37.933241 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:39.832232 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:39.832148 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:40.613254 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:40.613225 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:40.767002 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:40.766971 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:40.925847 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:40.925771 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:41.722909 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:41.722866 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:43.140642 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:09:43.140608 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:09:43.814571 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:43.814532 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:43.958630 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:43.958597 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:44.110324 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:44.110247 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:44.528126 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:44.528093 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:45.559289 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:45.559256 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:45.744327 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:45.744296 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:45.898786 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:45.898696 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:46.098354 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:46.098321 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:46.275916 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:46.275881 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:47.802576 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:47.802535 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:47.954978 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:47.954944 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:48.921399 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:48.921362 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:49.064547 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:49.064513 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:49.927300 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:49.927267 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:51.716303 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:51.716268 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:51.872760 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:51.872726 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:52.034059 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:52.034020 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:52.947407 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:52.947371 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:53.098217 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:53.098154 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:53.277429 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:53.277392 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:53.736223 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:53.736147 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:55.524737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:55.524693 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:57.371761 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:57.371724 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:58.967725 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:58.967691 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:59.134505 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:59.134475 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:59.776357 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:59.776321 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:09:59.945510 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:09:59.945475 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:00.110983 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:00.110888 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:02.249442 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:02.249399 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:03.091348 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:03.091312 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:03.246109 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:03.246075 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:04.163663 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:04.163626 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:04.326747 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:04.326704 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:06.184081 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:06.184049 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:06.329222 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:06.329147 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:08.048903 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:08.048869 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:09.859226 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:09.859193 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:10.509036 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:10.508992 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:10.658285 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:10.658237 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:11.912235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:11.912184 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:13.142876 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:10:13.142837 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:10:13.258692 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:13.258648 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:13.456674 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:13.456592 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:13.614229 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:13.614146 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:13.944227 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:13.944191 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:14.550805 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:14.550770 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:14.697746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:14.697666 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:15.763423 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:15.763389 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:16.003261 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:16.003220 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:18.173346 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:18.173238 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:19.615357 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:19.615319 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz"] Nov 02 08:10:19.615775 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:19.615530 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" podUID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" containerName="istio-proxy" containerID="cri-o://e1642f1d69a26da67af668eaa142699cafaa07ecdb5bba36ab666bf0985c7fbc" gracePeriod=30 Nov 02 08:10:20.022470 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:20.022437 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:21.881139 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:21.881099 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:22.052679 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:22.052646 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:22.215387 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:22.215297 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:23.542434 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.542396 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:23.694243 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.694202 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:23.694798 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.694772 5899 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:10:23.694866 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.694831 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" podUID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:10:23.804867 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.804770 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh"] Nov 02 08:10:23.809845 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.809816 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.836406 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.836372 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh"] Nov 02 08:10:23.854002 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.853965 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r8mr\" (UniqueName: \"kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-kube-api-access-4r8mr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.854238 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.854025 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.854238 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.854186 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.854374 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.854252 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.854374 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.854329 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.854463 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.854380 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:23.854463 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.854386 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9386adea-631b-4d70-a3f6-243b4ce8836d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.854463 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.854447 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.854589 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.854485 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.854589 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.854501 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.955535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.955498 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8mr\" (UniqueName: \"kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-kube-api-access-4r8mr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.955743 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.955542 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.955743 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.955620 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.955743 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.955699 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.955743 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.955733 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.955969 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.955766 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9386adea-631b-4d70-a3f6-243b4ce8836d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.955969 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.955810 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.955969 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.955840 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.956225 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.956180 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.956454 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.956197 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.956551 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.956227 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.956613 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.956460 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.956817 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.956792 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.957068 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.957051 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9386adea-631b-4d70-a3f6-243b4ce8836d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.958933 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.958908 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.959030 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.959000 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.985692 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.985658 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:23.985916 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:23.985891 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r8mr\" (UniqueName: \"kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-kube-api-access-4r8mr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxxrh\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:24.018899 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:24.018867 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:24.131185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:24.131086 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:24.199451 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:24.197934 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:24.462070 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:24.462040 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh"] Nov 02 08:10:24.463442 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:10:24.463410 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9386adea_631b_4d70_a3f6_243b4ce8836d.slice/crio-2ab8c9560d71701b90d8f3b28a666cd6e7727bc871b520c042e37789bba00223 WatchSource:0}: Error finding container 2ab8c9560d71701b90d8f3b28a666cd6e7727bc871b520c042e37789bba00223: Status 404 returned error can't find the container with id 2ab8c9560d71701b90d8f3b28a666cd6e7727bc871b520c042e37789bba00223 Nov 02 08:10:24.466342 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:24.466301 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:10:24.466426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:24.466380 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:10:24.466426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:24.466410 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:10:24.704112 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:24.704061 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" event={"ID":"9386adea-631b-4d70-a3f6-243b4ce8836d","Type":"ContainerStarted","Data":"a3e273a8f0aca0b4007f3d201116c9c799cc18779e454cccaef35b00fc36ad4d"} Nov 02 08:10:24.704799 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:24.704205 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" event={"ID":"9386adea-631b-4d70-a3f6-243b4ce8836d","Type":"ContainerStarted","Data":"2ab8c9560d71701b90d8f3b28a666cd6e7727bc871b520c042e37789bba00223"} Nov 02 08:10:24.710611 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:24.710528 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:24.756568 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:24.756512 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" podStartSLOduration=1.756497244 podStartE2EDuration="1.756497244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:24.754391551 +0000 UTC m=+2725.052672377" watchObservedRunningTime="2026-11-02 08:10:24.756497244 +0000 UTC m=+2725.054778069" Nov 02 08:10:24.855458 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:24.855421 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:24.946383 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:24.946351 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 08:10:25.110183 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.105594 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-kube-api-access-hb9jg\" (UniqueName: \"kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-kube-api-access-hb9jg\") pod \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " Nov 02 08:10:25.110183 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.105646 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istiod-ca-cert\") pod \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " Nov 02 08:10:25.110183 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.105699 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-certs\") pod \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " Nov 02 08:10:25.110183 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.105726 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-data\") pod \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " Nov 02 08:10:25.110183 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.105804 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-envoy\") pod \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " Nov 02 08:10:25.110183 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.105979 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-socket\") pod \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " Nov 02 08:10:25.110183 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.106015 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-podinfo\") pod \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " Nov 02 08:10:25.110183 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.106055 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-credential-socket\") pod \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " Nov 02 08:10:25.110183 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.106101 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-token\" (UniqueName: \"kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-token\") pod \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\" (UID: \"e12bb3bc-8ded-4004-a7c3-625003bf4e74\") " Nov 02 08:10:25.110183 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.107461 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-data" pod "e12bb3bc-8ded-4004-a7c3-625003bf4e74" (UID: "e12bb3bc-8ded-4004-a7c3-625003bf4e74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:25.110936 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.110711 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-token" pod "e12bb3bc-8ded-4004-a7c3-625003bf4e74" (UID: "e12bb3bc-8ded-4004-a7c3-625003bf4e74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:25.127190 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.115343 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istiod-ca-cert" pod "e12bb3bc-8ded-4004-a7c3-625003bf4e74" (UID: "e12bb3bc-8ded-4004-a7c3-625003bf4e74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:10:25.137746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.129591 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-credential-socket" pod "e12bb3bc-8ded-4004-a7c3-625003bf4e74" (UID: "e12bb3bc-8ded-4004-a7c3-625003bf4e74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:25.137746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.129794 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-certs" pod "e12bb3bc-8ded-4004-a7c3-625003bf4e74" (UID: "e12bb3bc-8ded-4004-a7c3-625003bf4e74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:25.137746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.129860 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-socket" pod "e12bb3bc-8ded-4004-a7c3-625003bf4e74" (UID: "e12bb3bc-8ded-4004-a7c3-625003bf4e74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:25.137746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.131963 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:25.139257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.138368 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxxrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 08:10:25.139257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.138461 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" podUID="9386adea-631b-4d70-a3f6-243b4ce8836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 08:10:25.139910 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.139550 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-podinfo" pod "e12bb3bc-8ded-4004-a7c3-625003bf4e74" (UID: "e12bb3bc-8ded-4004-a7c3-625003bf4e74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:10:25.142237 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.142209 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-envoy" pod "e12bb3bc-8ded-4004-a7c3-625003bf4e74" (UID: "e12bb3bc-8ded-4004-a7c3-625003bf4e74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:10:25.142675 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.142643 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-kube-api-access-hb9jg" pod "e12bb3bc-8ded-4004-a7c3-625003bf4e74" (UID: "e12bb3bc-8ded-4004-a7c3-625003bf4e74"). InnerVolumeSpecName "kube-api-access-hb9jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:10:25.207630 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.207579 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb9jg\" (UniqueName: \"kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-kube-api-access-hb9jg\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:10:25.207630 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.207613 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:10:25.207630 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.207623 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:10:25.207630 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.207633 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:10:25.207630 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.207641 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:10:25.208056 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.207651 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:10:25.208056 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.207663 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:10:25.208056 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.207676 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12bb3bc-8ded-4004-a7c3-625003bf4e74-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:10:25.208056 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.207688 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12bb3bc-8ded-4004-a7c3-625003bf4e74-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:10:25.711353 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.711319 5899 generic.go:356] "Generic (PLEG): container finished" podID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" containerID="e1642f1d69a26da67af668eaa142699cafaa07ecdb5bba36ab666bf0985c7fbc" exitCode=0 Nov 02 08:10:25.711825 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.711372 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" Nov 02 08:10:25.711825 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.711404 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" event={"ID":"e12bb3bc-8ded-4004-a7c3-625003bf4e74","Type":"ContainerDied","Data":"e1642f1d69a26da67af668eaa142699cafaa07ecdb5bba36ab666bf0985c7fbc"} Nov 02 08:10:25.711825 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.711460 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz" event={"ID":"e12bb3bc-8ded-4004-a7c3-625003bf4e74","Type":"ContainerDied","Data":"5fe3b7db6cba14fc8c06f8d0af9f596e47d50e68817e1dbfd9306b9c41493b97"} Nov 02 08:10:25.711825 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.711497 5899 scope.go:122] "RemoveContainer" containerID="e1642f1d69a26da67af668eaa142699cafaa07ecdb5bba36ab666bf0985c7fbc" Nov 02 08:10:25.725728 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.725705 5899 scope.go:122] "RemoveContainer" containerID="e1642f1d69a26da67af668eaa142699cafaa07ecdb5bba36ab666bf0985c7fbc" Nov 02 08:10:25.726088 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:10:25.726066 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1642f1d69a26da67af668eaa142699cafaa07ecdb5bba36ab666bf0985c7fbc\": container with ID starting with e1642f1d69a26da67af668eaa142699cafaa07ecdb5bba36ab666bf0985c7fbc not found: ID does not exist" containerID="e1642f1d69a26da67af668eaa142699cafaa07ecdb5bba36ab666bf0985c7fbc" Nov 02 08:10:25.726152 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.726101 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1642f1d69a26da67af668eaa142699cafaa07ecdb5bba36ab666bf0985c7fbc"} err="failed to get container status \"e1642f1d69a26da67af668eaa142699cafaa07ecdb5bba36ab666bf0985c7fbc\": rpc error: code = NotFound desc = could not find container \"e1642f1d69a26da67af668eaa142699cafaa07ecdb5bba36ab666bf0985c7fbc\": container with ID starting with e1642f1d69a26da67af668eaa142699cafaa07ecdb5bba36ab666bf0985c7fbc not found: ID does not exist" Nov 02 08:10:25.781234 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.781143 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz"] Nov 02 08:10:25.791772 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.791743 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-dbtkz"] Nov 02 08:10:25.830477 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:25.830439 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" path="/var/lib/kubelet/pods/e12bb3bc-8ded-4004-a7c3-625003bf4e74/volumes" Nov 02 08:10:26.131574 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:26.131530 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxxrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 08:10:26.131783 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:26.131598 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" podUID="9386adea-631b-4d70-a3f6-243b4ce8836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 08:10:26.287678 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:26.287632 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:27.131609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:27.131576 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxxrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 08:10:27.132025 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:27.131635 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" podUID="9386adea-631b-4d70-a3f6-243b4ce8836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 08:10:27.385721 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:27.385628 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:28.131991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:28.131958 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxxrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 08:10:28.132409 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:28.132014 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" podUID="9386adea-631b-4d70-a3f6-243b4ce8836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 08:10:28.551901 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:28.551863 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:28.710876 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:28.710841 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:29.132289 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:29.132251 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxxrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 08:10:29.132701 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:29.132317 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" podUID="9386adea-631b-4d70-a3f6-243b4ce8836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 08:10:30.132238 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:30.132197 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxxrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 08:10:30.132391 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:30.132271 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" podUID="9386adea-631b-4d70-a3f6-243b4ce8836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 08:10:30.700464 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:30.700412 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:31.132013 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:31.131981 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxxrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 08:10:31.132213 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:31.132033 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" podUID="9386adea-631b-4d70-a3f6-243b4ce8836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 08:10:32.132201 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:32.132144 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxxrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 08:10:32.132609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:32.132245 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" podUID="9386adea-631b-4d70-a3f6-243b4ce8836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 08:10:32.718637 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:32.718600 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:33.271352 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:33.271295 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:33.271916 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:33.271893 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:33.272725 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:33.272706 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:10:34.655362 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:34.655326 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:36.729578 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:36.729546 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:37.905245 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:37.905214 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:38.726553 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.726522 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:38.872360 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.872321 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq"] Nov 02 08:10:38.873142 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.873111 5899 cpu_manager.go:414] "containerMap: removing container" podUID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" containerName="istio-proxy" Nov 02 08:10:38.873142 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.873139 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" containerName="istio-proxy" Nov 02 08:10:38.873351 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.873316 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12bb3bc-8ded-4004-a7c3-625003bf4e74" containerName="istio-proxy" Nov 02 08:10:38.877249 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.877223 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:38.911969 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.911932 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq"] Nov 02 08:10:38.919306 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.919268 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:38.919487 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.919389 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:38.919562 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.919492 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:38.919562 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.919516 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880640a1-42c9-4be2-8b81-c7af629b8a50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:38.919750 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.919725 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blnxk\" (UniqueName: \"kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-kube-api-access-blnxk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:38.919909 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.919887 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:38.919996 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.919970 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:38.920079 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.920061 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:38.920123 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:38.920113 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.021241 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.021196 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.021241 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.021247 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.021535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.021285 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.021535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.021312 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.021535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.021363 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.021535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.021401 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.021535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.021464 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.021535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.021490 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880640a1-42c9-4be2-8b81-c7af629b8a50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.021855 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.021568 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blnxk\" (UniqueName: \"kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-kube-api-access-blnxk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.021855 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.021699 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.021972 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.021864 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.021972 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.021963 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.022077 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.022023 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.022445 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.022422 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880640a1-42c9-4be2-8b81-c7af629b8a50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.023761 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.023739 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.023761 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.023749 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.073956 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.073917 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blnxk\" (UniqueName: \"kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-kube-api-access-blnxk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.074104 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.073916 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f6dgq\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.195973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.195939 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:39.527046 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.527019 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq"] Nov 02 08:10:39.528034 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:10:39.528008 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880640a1_42c9_4be2_8b81_c7af629b8a50.slice/crio-598d1624af0153cb0f76a2488a209fb60108ed74f7251673c00c8aa2c012a1cc WatchSource:0}: Error finding container 598d1624af0153cb0f76a2488a209fb60108ed74f7251673c00c8aa2c012a1cc: Status 404 returned error can't find the container with id 598d1624af0153cb0f76a2488a209fb60108ed74f7251673c00c8aa2c012a1cc Nov 02 08:10:39.530762 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.530716 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:10:39.530880 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.530805 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:10:39.530880 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.530834 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:10:39.825950 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.825866 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" event={"ID":"880640a1-42c9-4be2-8b81-c7af629b8a50","Type":"ContainerStarted","Data":"725b7097307d9fac98c6aabf2ccb6befed2882ce6f0b0fdef3bdba829faf0f44"} Nov 02 08:10:39.825950 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.825927 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" event={"ID":"880640a1-42c9-4be2-8b81-c7af629b8a50","Type":"ContainerStarted","Data":"598d1624af0153cb0f76a2488a209fb60108ed74f7251673c00c8aa2c012a1cc"} Nov 02 08:10:39.882456 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:39.882405 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" podStartSLOduration=1.882391434 podStartE2EDuration="1.882391434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:39.879777095 +0000 UTC m=+2740.178057920" watchObservedRunningTime="2026-11-02 08:10:39.882391434 +0000 UTC m=+2740.180672259" Nov 02 08:10:40.196444 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:40.196358 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:40.198399 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:40.198371 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f6dgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:10:40.198557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:40.198426 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" podUID="880640a1-42c9-4be2-8b81-c7af629b8a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 08:10:40.839396 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:40.839366 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:41.196900 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:41.196796 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f6dgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:10:41.196900 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:41.196867 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" podUID="880640a1-42c9-4be2-8b81-c7af629b8a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 08:10:42.201181 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:42.196575 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f6dgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:10:42.201181 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:42.196661 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" podUID="880640a1-42c9-4be2-8b81-c7af629b8a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 08:10:43.142563 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:10:43.142522 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:10:43.196389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:43.196349 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f6dgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:10:43.196670 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:43.196643 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" podUID="880640a1-42c9-4be2-8b81-c7af629b8a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 08:10:43.412640 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:43.412551 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:44.205304 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:44.205266 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f6dgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:10:44.205640 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:44.205598 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" podUID="880640a1-42c9-4be2-8b81-c7af629b8a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 08:10:45.196952 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:45.196515 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f6dgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:10:45.196952 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:45.196575 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" podUID="880640a1-42c9-4be2-8b81-c7af629b8a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 08:10:45.902553 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:45.902508 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:46.197263 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:46.197176 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f6dgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:10:46.197263 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:46.197240 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" podUID="880640a1-42c9-4be2-8b81-c7af629b8a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 08:10:47.196225 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:47.196185 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f6dgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 08:10:47.196399 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:47.196246 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" podUID="880640a1-42c9-4be2-8b81-c7af629b8a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 08:10:47.950079 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:47.950042 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:48.397869 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:48.397831 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:48.398384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:48.398354 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:48.399229 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:48.399205 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:10:50.166270 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:50.166235 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:52.153649 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:52.153615 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:54.046920 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:54.046876 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:56.140978 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:56.140936 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:10:58.140714 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:10:58.140670 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:00.725880 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:00.725840 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:02.492309 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:02.492264 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:04.306643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:04.306609 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:06.333517 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:06.333479 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:06.464235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:06.464190 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:08.027088 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:08.027050 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:09.812124 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:09.812054 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:11.767656 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:11.767609 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:13.141404 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:11:13.141358 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:11:13.484005 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:13.483969 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:15.359450 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:15.359412 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:17.095962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:17.095896 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:18.966542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:18.966497 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:20.714804 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:20.714725 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:22.411975 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:22.411902 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:24.352653 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:24.352602 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:26.148769 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:26.148739 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:27.899286 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:27.899248 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:29.606285 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:29.606232 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:31.339459 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:31.339379 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:31.518815 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:31.518777 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:31.657450 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:31.657360 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:33.101287 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:33.101250 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:34.987973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:34.987942 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:35.123782 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:35.123748 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:35.736020 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:35.735970 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:35.870360 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:35.870330 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:36.777736 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:36.777680 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:38.578642 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:38.578608 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:40.353476 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:40.353440 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:42.225446 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:42.225409 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:43.141042 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:11:43.141006 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:11:43.594844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:43.594801 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:43.733714 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:43.733680 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:44.337374 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:44.337334 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:44.582338 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:44.582301 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:46.314003 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:46.313938 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:47.989659 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:47.989625 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:49.654034 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:49.653968 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:51.414203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:51.414157 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:53.288079 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:53.288035 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:55.562369 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:55.562337 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:57.305633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:57.305600 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:11:59.110025 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:11:59.109995 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:00.878332 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:00.878294 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:02.566599 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:02.566567 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:04.398562 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:04.398528 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:06.093188 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:06.093133 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:06.499365 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:06.499328 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:07.755412 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:07.755374 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:08.879955 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:08.879902 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:09.018361 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:09.018325 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:09.717151 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:09.717115 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:10.175730 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:10.175692 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:10.316435 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:10.316402 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:10.538748 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:10.538712 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:10.831639 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:10.831550 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:10.977370 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:10.977335 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:11.166050 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:11.165967 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:11.810439 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:11.810399 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:11.947839 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:11.947803 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:12.080881 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:12.080794 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:13.150765 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:12:13.150715 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:12:13.221797 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:13.221764 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:13.378102 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:13.378056 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:13.522393 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:13.522355 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:14.452352 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:14.452320 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:14.603475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:14.603435 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:15.521410 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:15.521368 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:16.697451 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:16.697415 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:18.485071 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:18.485036 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:18.619603 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:18.619565 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:18.930229 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:18.930104 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:20.458353 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:20.458320 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:22.210111 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:22.210076 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:23.947756 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:23.947726 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:24.721504 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:24.721473 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:24.854486 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:24.854455 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:25.396069 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:25.396029 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:26.063059 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:26.062994 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:27.890705 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:27.890666 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:29.620217 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:29.620152 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:31.202862 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:31.202825 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:31.350036 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:31.350004 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:31.575828 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:31.575790 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:32.500348 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:32.500307 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:32.655562 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:32.655518 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:32.797031 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:32.796928 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:33.091253 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:33.091130 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:33.452290 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:33.452201 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:33.714920 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:33.714842 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:33.850527 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:33.850494 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:35.530358 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:35.530323 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:36.587183 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:36.586607 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:36.876096 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:36.876004 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:38.931781 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:38.931753 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:40.340430 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:40.340393 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:40.594658 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:40.594565 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:40.735024 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:40.734984 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:42.527022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:42.526946 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:43.144194 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:12:43.144134 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:12:44.271826 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:44.271798 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:45.748292 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:45.748259 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz"] Nov 02 08:12:45.752593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:45.752570 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" Nov 02 08:12:45.770694 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:45.770670 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz"] Nov 02 08:12:45.780443 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:45.780411 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-ae4260\"/\"kube-root-ca.crt\"" Nov 02 08:12:45.784783 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:45.784758 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-ae4260\"/\"default-dockercfg-db5vb\"" Nov 02 08:12:45.784932 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:45.784828 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-ae4260\"/\"openshift-service-ca.crt\"" Nov 02 08:12:45.784932 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:45.784839 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-ae4260\"/\"e2e-usage-sim-b65a81-cert\"" Nov 02 08:12:45.876944 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:45.876909 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ptlg\" (UniqueName: \"kubernetes.io/projected/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-kube-api-access-5ptlg\") pod \"e2e-usage-sim-b65a81-97c94d6c7-vkcrz\" (UID: \"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc\") " pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" Nov 02 08:12:45.876944 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:45.876951 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-tls\") pod \"e2e-usage-sim-b65a81-97c94d6c7-vkcrz\" (UID: \"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc\") " pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" Nov 02 08:12:45.978873 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:45.978834 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ptlg\" (UniqueName: \"kubernetes.io/projected/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-kube-api-access-5ptlg\") pod \"e2e-usage-sim-b65a81-97c94d6c7-vkcrz\" (UID: \"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc\") " pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" Nov 02 08:12:45.978873 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:45.978878 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-tls\") pod \"e2e-usage-sim-b65a81-97c94d6c7-vkcrz\" (UID: \"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc\") " pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" Nov 02 08:12:45.981375 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:45.981349 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-tls\") pod \"e2e-usage-sim-b65a81-97c94d6c7-vkcrz\" (UID: \"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc\") " pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" Nov 02 08:12:46.002833 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:46.002758 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ptlg\" (UniqueName: \"kubernetes.io/projected/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-kube-api-access-5ptlg\") pod \"e2e-usage-sim-b65a81-97c94d6c7-vkcrz\" (UID: \"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc\") " pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" Nov 02 08:12:46.006469 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:46.006446 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:46.065741 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:46.065705 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" Nov 02 08:12:46.370286 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:46.370259 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz"] Nov 02 08:12:46.371897 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:12:46.371871 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e92c0c4_0c58_4a4b_ac3e_7e7ffe4245bc.slice/crio-5f56b5940015d57296049e2c28d340720f953c031697b99ecb638c8a8351c2ef WatchSource:0}: Error finding container 5f56b5940015d57296049e2c28d340720f953c031697b99ecb638c8a8351c2ef: Status 404 returned error can't find the container with id 5f56b5940015d57296049e2c28d340720f953c031697b99ecb638c8a8351c2ef Nov 02 08:12:46.776006 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:46.775970 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" event={"ID":"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc","Type":"ContainerStarted","Data":"e06358da4aa8cbb0f6a322251451c0e52213fa6293d7ea6001f82fbc36d971ac"} Nov 02 08:12:46.776456 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:46.776014 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" event={"ID":"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc","Type":"ContainerStarted","Data":"5f56b5940015d57296049e2c28d340720f953c031697b99ecb638c8a8351c2ef"} Nov 02 08:12:46.776456 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:46.776178 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" Nov 02 08:12:46.781324 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:46.781300 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" Nov 02 08:12:46.818847 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:46.818797 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" podStartSLOduration=1.818784948 podStartE2EDuration="1.818784948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:46.81720816 +0000 UTC m=+2867.115488986" watchObservedRunningTime="2026-11-02 08:12:46.818784948 +0000 UTC m=+2867.117065804" Nov 02 08:12:47.753870 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:47.753838 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:49.515480 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:49.515445 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:51.320368 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:51.320325 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:51.721621 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:51.721489 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:51.886592 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:51.886553 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:53.562622 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:53.562587 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:55.341725 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:55.341692 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:57.163806 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:57.163769 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:12:58.896294 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:12:58.896258 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:00.576777 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:00.576744 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:02.312999 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:02.312964 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:04.114306 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:04.114269 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:05.726070 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:05.726035 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:06.633016 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:06.632982 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:07.600998 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:07.600968 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:09.352034 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:09.351999 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:11.130405 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:11.130334 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:12.707009 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:12.706972 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:12.843149 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:12.843087 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:13.143770 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:13:13.143726 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:13:13.155403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:13.155365 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:13.831246 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:13.831202 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:13.968993 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:13.968958 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:14.408191 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:14.408130 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:15.099039 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:15.099007 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:16.911638 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:16.911609 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:18.612807 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:18.612768 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:20.387526 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:20.387487 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:21.000782 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:21.000742 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:21.138335 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:21.138302 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:21.273486 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:21.273377 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:22.102471 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:22.102436 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:23.904255 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:23.904209 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:25.710333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:25.710302 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:27.495829 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:27.495797 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:27.634540 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:27.634502 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:27.778789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:27.778693 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:28.091927 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:28.091836 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:28.547396 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:28.547359 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:28.697506 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:28.697423 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:29.318135 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:29.318104 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:30.840253 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:30.840214 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:30.989540 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:30.989483 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:31.504908 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:31.504874 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:31.978024 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:31.977993 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:32.122872 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:32.122839 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:33.389348 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:33.389317 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:33.704129 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:33.704043 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:33.846154 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:33.846109 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:35.210814 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:35.210778 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:36.913261 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:36.913228 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:38.603730 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:38.603697 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:40.480500 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:40.480469 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:42.210131 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:42.210071 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:42.804830 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:42.804796 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:42.950656 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:42.950617 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:43.229724 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:13:43.229649 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:13:43.421362 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:43.421329 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:44.210790 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:44.210760 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:45.689795 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:45.689757 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:45.828839 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:45.828800 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:45.965542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:45.965403 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:47.618487 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:47.618437 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:49.241900 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:49.241847 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:49.385130 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:49.385096 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:49.519224 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:49.519131 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:49.910704 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:49.909192 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:50.200823 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:50.200731 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:50.365037 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:50.365003 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:50.494192 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:50.494124 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:50.673511 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:50.673474 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:51.522210 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:51.522137 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:52.499907 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:52.499869 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:52.645852 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:52.645813 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:54.007001 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:54.006964 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:54.314091 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:54.313999 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:55.400826 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:55.400794 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:55.953395 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:55.953362 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:57.697433 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:57.697399 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:58.858686 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:58.858633 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:58.988928 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:58.988894 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:59.384868 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:59.384836 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:59.518434 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:59.518400 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:13:59.659257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:13:59.659133 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:01.033751 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:01.033715 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:01.414338 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:01.414260 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:03.165012 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:03.164978 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:04.510609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:04.510567 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:04.901088 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:04.901005 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:05.422929 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:05.422896 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:05.570024 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:05.569985 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:05.855584 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:05.855550 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:06.326230 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:06.326197 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:06.461641 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:06.461580 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:06.929580 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:06.929525 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:07.057535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:07.057498 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:07.197598 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:07.197466 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:08.783355 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:08.783310 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:10.264602 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:10.264567 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:10.415592 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:10.415556 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:10.635067 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:10.634984 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:11.997944 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:11.997908 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:12.147353 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:12.147314 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:12.548093 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:12.548058 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:13.146201 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:14:13.144668 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:14:14.376827 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:14.376759 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:16.249547 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.249502 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb"] Nov 02 08:14:16.254736 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.254709 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.283980 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.283948 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e271b1-serving-cert\"" Nov 02 08:14:16.287960 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.287929 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb"] Nov 02 08:14:16.346573 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.346538 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-metrics-certs\") pod \"maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.346755 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.346736 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.346817 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.346760 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zscjb\" (UniqueName: \"kubernetes.io/projected/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-kube-api-access-zscjb\") pod \"maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.346915 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.346895 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.364895 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.364867 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:16.448552 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.448520 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.448552 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.448555 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zscjb\" (UniqueName: \"kubernetes.io/projected/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-kube-api-access-zscjb\") pod \"maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.448815 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.448620 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.448815 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.448682 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-metrics-certs\") pod \"maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.449423 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.449395 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.451070 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.451048 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-maas-api-tls\") pod \"maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.451226 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.451205 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-metrics-certs\") pod \"maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.472471 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.472448 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zscjb\" (UniqueName: \"kubernetes.io/projected/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-kube-api-access-zscjb\") pod \"maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.498748 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.498711 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:16.568617 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.568523 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:16.635037 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.634718 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:16.871544 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.871520 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb"] Nov 02 08:14:16.873223 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:14:16.873198 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6cad1f_c205_4e36_8c0a_3d9dd8d97bf5.slice/crio-ec69e52c34e50c923fe27500678d2d8e49f363094a0858db60364151ece04cc3 WatchSource:0}: Error finding container ec69e52c34e50c923fe27500678d2d8e49f363094a0858db60364151ece04cc3: Status 404 returned error can't find the container with id ec69e52c34e50c923fe27500678d2d8e49f363094a0858db60364151ece04cc3 Nov 02 08:14:16.874530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:16.874509 5899 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:14:17.444722 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:17.444633 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" event={"ID":"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5","Type":"ContainerStarted","Data":"ec69e52c34e50c923fe27500678d2d8e49f363094a0858db60364151ece04cc3"} Nov 02 08:14:18.157949 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:18.157903 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:18.453257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:18.453149 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" event={"ID":"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5","Type":"ContainerStarted","Data":"3f774c96518934619fa47985f44fa47e177e9c4bb8e747abc94e1c235dfadbd3"} Nov 02 08:14:18.453257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:18.453240 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:18.500881 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:18.500823 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" podStartSLOduration=2.009016704 podStartE2EDuration="2.500810392s" totalImagesPullingTime="491.793688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:14:18.496576968 +0000 UTC m=+2958.794857787" watchObservedRunningTime="2026-11-02 08:14:18.500810392 +0000 UTC m=+2958.799091597" Nov 02 08:14:19.984496 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:19.984462 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:21.719416 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:21.719378 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:23.502142 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:23.502103 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:24.465129 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:24.465098 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:14:25.236537 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:25.236502 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:26.953914 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:26.953880 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:28.761569 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:28.761535 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:30.422298 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:30.422263 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:32.106967 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:32.106932 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:33.813597 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:33.813566 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:35.509509 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:35.509479 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:37.168366 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:37.168335 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:38.947287 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:38.947249 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:40.605349 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:40.605315 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:42.404095 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:42.404058 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:43.141099 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:14:43.141063 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:14:44.137508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:44.137471 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:45.965881 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:45.965850 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:47.735115 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:47.735083 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:49.638700 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:49.638667 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:51.465181 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:51.465118 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:53.230265 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:53.230232 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:54.969553 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:54.969520 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:55.113452 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:55.113413 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:55.252735 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:55.252685 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:56.856469 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:56.856440 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:14:58.749723 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:14:58.749675 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:00.744994 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:00.744959 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:02.672897 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:02.672862 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:04.681482 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:04.681446 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:06.525217 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:06.525186 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:07.054402 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:07.054362 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:07.234028 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:07.233993 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:08.395742 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:08.395709 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:08.987005 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:08.986973 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:09.356408 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:09.356321 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:09.924646 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:09.924604 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:10.084928 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:10.084894 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:10.224554 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:10.224464 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:12.023270 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:12.023230 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:13.143476 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:15:13.143446 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:15:13.489322 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:13.489289 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:13.655614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:13.655581 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:14.054803 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:14.054770 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:14.741645 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:14.741614 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:14.886145 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:14.886110 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:15.268930 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:15.268837 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:15.414310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:15.414277 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:16.228545 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:16.228512 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:18.147055 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:18.147025 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:19.964807 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:19.964769 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:22.005842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:22.005811 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:22.518109 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:22.518066 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:22.649516 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:22.649473 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:24.102807 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:24.102768 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:25.782874 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:25.782839 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:27.587465 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:27.587420 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:29.319468 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:29.319428 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:30.993419 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:30.993358 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:32.664325 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:32.664289 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:34.420343 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:34.420278 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:36.103708 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:36.103658 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:37.780036 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:37.780001 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:39.522436 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:39.522400 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:41.322591 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:41.322549 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:43.107830 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:43.107791 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:43.149736 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:15:43.149692 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:15:44.977646 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:44.977605 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:45.668313 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:45.668282 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:45.810290 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:45.810258 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:46.233786 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:46.233750 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:46.369616 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:46.369583 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:47.096948 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:47.096913 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:47.291960 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:47.291928 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:47.425573 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:47.425472 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:47.793781 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:47.793741 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:48.267959 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:48.267921 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:48.413957 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:48.413923 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:48.926822 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:48.926786 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:50.907484 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:50.907450 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:51.848881 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:51.848833 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:51.992532 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:51.992496 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:52.847668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:52.847625 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:52.998407 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:52.998375 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:53.136405 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:53.136319 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:53.312701 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:53.312669 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:53.636351 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:53.636311 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:53.776891 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:53.776817 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:53.971441 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:53.969988 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:54.164481 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:54.164446 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:55.104303 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:55.104272 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:56.980362 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:56.980323 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:57.863030 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:57.862994 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:58.015621 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:58.015582 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:58.906177 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:58.906122 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:59.590634 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:59.590595 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:15:59.728867 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:15:59.728832 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:00.930586 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:00.930548 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:02.558915 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.558874 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669"] Nov 02 08:16:02.563301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.563282 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.584920 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.584892 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669"] Nov 02 08:16:02.593499 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.593469 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f765b8-serving-cert\"" Nov 02 08:16:02.685556 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.685520 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:02.709541 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.709505 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxzl\" (UniqueName: \"kubernetes.io/projected/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-kube-api-access-mmxzl\") pod \"maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.709743 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.709653 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-metrics-certs\") pod \"maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.709743 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.709711 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-maas-api-tls\") pod \"maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.709837 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.709747 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.810899 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.810814 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-metrics-certs\") pod \"maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.810899 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.810862 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-maas-api-tls\") pod \"maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.810899 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.810898 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.811206 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.810993 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxzl\" (UniqueName: \"kubernetes.io/projected/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-kube-api-access-mmxzl\") pod \"maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.811632 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.811604 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.813370 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.813347 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-maas-api-tls\") pod \"maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.813483 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.813373 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-metrics-certs\") pod \"maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.821303 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.821268 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:02.835415 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.835383 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxzl\" (UniqueName: \"kubernetes.io/projected/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-kube-api-access-mmxzl\") pod \"maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.877250 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.877216 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:02.962213 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:02.962130 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:03.186302 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:03.186276 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669"] Nov 02 08:16:03.187905 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:16:03.187875 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4269e0_a8c4_40fa_90c6_34a67ac7e810.slice/crio-5ae60f64192b18dc937121c4f4fd0bfe8b8c954d19acfc4c0aa391eb54a03150 WatchSource:0}: Error finding container 5ae60f64192b18dc937121c4f4fd0bfe8b8c954d19acfc4c0aa391eb54a03150: Status 404 returned error can't find the container with id 5ae60f64192b18dc937121c4f4fd0bfe8b8c954d19acfc4c0aa391eb54a03150 Nov 02 08:16:04.166918 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:04.166879 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" event={"ID":"8e4269e0-a8c4-40fa-90c6-34a67ac7e810","Type":"ContainerStarted","Data":"a3abf0bf4c3798fe0b91650653f999e8920ca1c066a8dca9cbc28ceb7b372537"} Nov 02 08:16:04.167363 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:04.166949 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:04.167363 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:04.166971 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" event={"ID":"8e4269e0-a8c4-40fa-90c6-34a67ac7e810","Type":"ContainerStarted","Data":"5ae60f64192b18dc937121c4f4fd0bfe8b8c954d19acfc4c0aa391eb54a03150"} Nov 02 08:16:04.213516 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:04.213465 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" podStartSLOduration=1.810565137 podStartE2EDuration="2.213450967s" totalImagesPullingTime="402.88583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:04.210493898 +0000 UTC m=+3064.508774723" watchObservedRunningTime="2026-11-02 08:16:04.213450967 +0000 UTC m=+3064.511731792" Nov 02 08:16:04.688497 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:04.688463 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:06.523121 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:06.523084 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:07.177432 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:07.177397 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:08.397901 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:08.397860 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:10.123521 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:10.123489 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:10.179740 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:10.179711 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:16:11.897101 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:11.897022 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:13.141325 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:16:13.141290 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:16:13.629508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:13.629472 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:15.490050 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:15.490012 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:17.312310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:17.312271 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:17.456489 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:17.456452 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:19.391619 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:19.391584 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:20.424051 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:20.424014 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:20.568745 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:20.568705 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:20.944221 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:20.944183 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz"] Nov 02 08:16:20.944409 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:20.944375 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" podUID="6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc" containerName="sim" containerID="cri-o://e06358da4aa8cbb0f6a322251451c0e52213fa6293d7ea6001f82fbc36d971ac" gracePeriod=30 Nov 02 08:16:21.108353 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.108312 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:21.265932 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.265883 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:21.273951 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.273574 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" Nov 02 08:16:21.288750 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.288367 5899 generic.go:356] "Generic (PLEG): container finished" podID="6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc" containerID="e06358da4aa8cbb0f6a322251451c0e52213fa6293d7ea6001f82fbc36d971ac" exitCode=0 Nov 02 08:16:21.288750 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.288440 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" Nov 02 08:16:21.288750 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.288541 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" event={"ID":"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc","Type":"ContainerDied","Data":"e06358da4aa8cbb0f6a322251451c0e52213fa6293d7ea6001f82fbc36d971ac"} Nov 02 08:16:21.288750 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.288580 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz" event={"ID":"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc","Type":"ContainerDied","Data":"5f56b5940015d57296049e2c28d340720f953c031697b99ecb638c8a8351c2ef"} Nov 02 08:16:21.288750 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.288603 5899 scope.go:122] "RemoveContainer" containerID="e06358da4aa8cbb0f6a322251451c0e52213fa6293d7ea6001f82fbc36d971ac" Nov 02 08:16:21.307035 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.307008 5899 scope.go:122] "RemoveContainer" containerID="e06358da4aa8cbb0f6a322251451c0e52213fa6293d7ea6001f82fbc36d971ac" Nov 02 08:16:21.307568 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:16:21.307540 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06358da4aa8cbb0f6a322251451c0e52213fa6293d7ea6001f82fbc36d971ac\": container with ID starting with e06358da4aa8cbb0f6a322251451c0e52213fa6293d7ea6001f82fbc36d971ac not found: ID does not exist" containerID="e06358da4aa8cbb0f6a322251451c0e52213fa6293d7ea6001f82fbc36d971ac" Nov 02 08:16:21.307694 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.307577 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06358da4aa8cbb0f6a322251451c0e52213fa6293d7ea6001f82fbc36d971ac"} err="failed to get container status \"e06358da4aa8cbb0f6a322251451c0e52213fa6293d7ea6001f82fbc36d971ac\": rpc error: code = NotFound desc = could not find container \"e06358da4aa8cbb0f6a322251451c0e52213fa6293d7ea6001f82fbc36d971ac\": container with ID starting with e06358da4aa8cbb0f6a322251451c0e52213fa6293d7ea6001f82fbc36d971ac not found: ID does not exist" Nov 02 08:16:21.361205 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.361176 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-kube-api-access-5ptlg\" (UniqueName: \"kubernetes.io/projected/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-kube-api-access-5ptlg\") pod \"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc\" (UID: \"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc\") " Nov 02 08:16:21.361456 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.361438 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-tls\" (UniqueName: \"kubernetes.io/secret/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-tls\") pod \"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc\" (UID: \"6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc\") " Nov 02 08:16:21.366537 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.366509 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-kube-api-access-5ptlg" pod "6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc" (UID: "6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc"). InnerVolumeSpecName "kube-api-access-5ptlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:16:21.368614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.368556 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-tls" pod "6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc" (UID: "6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:16:21.464096 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.464010 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ptlg\" (UniqueName: \"kubernetes.io/projected/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-kube-api-access-5ptlg\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:16:21.464096 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.464050 5899 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:16:21.640883 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.640847 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz"] Nov 02 08:16:21.645623 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.645594 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-ae4260/e2e-usage-sim-b65a81-97c94d6c7-vkcrz"] Nov 02 08:16:21.820594 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:21.820556 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc" path="/var/lib/kubelet/pods/6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc/volumes" Nov 02 08:16:22.613497 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:22.613458 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:22.742789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:22.742734 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:23.012033 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:23.012002 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:23.210593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:23.210563 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:24.569747 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:24.569709 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:24.715320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:24.715284 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:25.565341 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:25.565302 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:26.471400 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:26.471366 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:28.237640 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:28.237608 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:30.074415 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:30.074382 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:30.214242 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:30.214212 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:30.634446 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:30.634411 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:31.105189 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:31.105142 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:31.270331 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:31.270293 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:31.902031 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:31.901997 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:33.711844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:33.711811 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:35.101983 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:35.101952 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:35.240746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:35.240713 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:35.690360 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:35.690327 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:36.859351 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:36.859309 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:36.995020 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:36.994985 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:37.712409 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:37.712329 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:39.497252 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:39.497212 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:40.773401 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:40.773344 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:40.912253 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:40.912215 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:41.351505 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.351470 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz"] Nov 02 08:16:41.352236 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.352218 5899 cpu_manager.go:414] "containerMap: removing container" podUID="6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc" containerName="sim" Nov 02 08:16:41.352236 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.352235 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc" containerName="sim" Nov 02 08:16:41.352400 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.352392 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e92c0c4-0c58-4a4b-ac3e-7e7ffe4245bc" containerName="sim" Nov 02 08:16:41.356255 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.356238 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.359388 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.359365 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-metrics-certs\") pod \"maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.359518 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.359495 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-maas-api-tls\") pod \"maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.359633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.359610 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctc8w\" (UniqueName: \"kubernetes.io/projected/181e9908-5351-46a9-9939-7559114bddd0-kube-api-access-ctc8w\") pod \"maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.359781 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.359761 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/181e9908-5351-46a9-9939-7559114bddd0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.372640 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.372590 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dfd6f0-serving-cert\"" Nov 02 08:16:41.391445 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.391406 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz"] Nov 02 08:16:41.427882 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.427850 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:41.460563 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.460530 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-metrics-certs\") pod \"maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.460730 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.460620 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-maas-api-tls\") pod \"maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.460730 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.460685 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctc8w\" (UniqueName: \"kubernetes.io/projected/181e9908-5351-46a9-9939-7559114bddd0-kube-api-access-ctc8w\") pod \"maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.460730 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.460704 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/181e9908-5351-46a9-9939-7559114bddd0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.460858 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:16:41.460751 5899 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-serving-cert: secret "maas-api-e2e-inf-b-dfd6f0-serving-cert" not found Nov 02 08:16:41.460858 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:16:41.460843 5899 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-maas-api-tls podName:181e9908-5351-46a9-9939-7559114bddd0 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:41.96081989 +0000 UTC m=+3102.259100705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-maas-api-tls") pod "maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" (UID: "181e9908-5351-46a9-9939-7559114bddd0") : secret "maas-api-e2e-inf-b-dfd6f0-serving-cert" not found Nov 02 08:16:41.461331 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.461314 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/181e9908-5351-46a9-9939-7559114bddd0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.463022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.463003 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-metrics-certs\") pod \"maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.485095 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.485068 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctc8w\" (UniqueName: \"kubernetes.io/projected/181e9908-5351-46a9-9939-7559114bddd0-kube-api-access-ctc8w\") pod \"maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.563770 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.563732 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:41.706918 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.706833 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:41.966106 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.966016 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-maas-api-tls\") pod \"maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.968530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.968509 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-maas-api-tls\") pod \"maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:41.970442 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:41.970424 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:42.276190 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:42.276147 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz"] Nov 02 08:16:42.277915 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:16:42.277876 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181e9908_5351_46a9_9939_7559114bddd0.slice/crio-81fe5725c1004f9f67a7b8237f55aa02ff909e41de695d5b446bd674dcd84ab0 WatchSource:0}: Error finding container 81fe5725c1004f9f67a7b8237f55aa02ff909e41de695d5b446bd674dcd84ab0: Status 404 returned error can't find the container with id 81fe5725c1004f9f67a7b8237f55aa02ff909e41de695d5b446bd674dcd84ab0 Nov 02 08:16:42.433509 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:42.433476 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" event={"ID":"181e9908-5351-46a9-9939-7559114bddd0","Type":"ContainerStarted","Data":"81fe5725c1004f9f67a7b8237f55aa02ff909e41de695d5b446bd674dcd84ab0"} Nov 02 08:16:42.526661 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:42.526579 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:43.141483 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:16:43.141445 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:16:43.325812 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:43.325782 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:43.441403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:43.441307 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" event={"ID":"181e9908-5351-46a9-9939-7559114bddd0","Type":"ContainerStarted","Data":"0c2d449d5c76e97935a8662f3622a873539e493d83c2bedcdef421168a1b22b9"} Nov 02 08:16:43.441556 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:43.441517 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:43.492584 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:43.492521 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" podStartSLOduration=2.163995164 podStartE2EDuration="2.492503441s" totalImagesPullingTime="328.508277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:43.489445299 +0000 UTC m=+3103.787726124" watchObservedRunningTime="2026-11-02 08:16:43.492503441 +0000 UTC m=+3103.790784266" Nov 02 08:16:45.028933 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:45.028897 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:46.792458 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:46.792408 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:47.445010 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:47.444931 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:47.590648 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:47.590619 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:47.915386 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:47.915331 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:48.394216 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.394144 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:48.495458 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.495425 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs"] Nov 02 08:16:48.504524 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.504489 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:48.517423 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.517384 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs"] Nov 02 08:16:48.525621 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.525589 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e6f4fe-serving-cert\"" Nov 02 08:16:48.568066 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.568025 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16310be4-c7d0-44de-bfee-579839e26705-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:48.568378 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.568352 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78dm8\" (UniqueName: \"kubernetes.io/projected/16310be4-c7d0-44de-bfee-579839e26705-kube-api-access-78dm8\") pod \"maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:48.568523 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.568507 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-maas-api-tls\") pod \"maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:48.568716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.568701 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-metrics-certs\") pod \"maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:48.585863 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.585818 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:48.670946 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.670846 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-metrics-certs\") pod \"maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:48.671136 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.671023 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16310be4-c7d0-44de-bfee-579839e26705-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:48.671136 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.671044 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78dm8\" (UniqueName: \"kubernetes.io/projected/16310be4-c7d0-44de-bfee-579839e26705-kube-api-access-78dm8\") pod \"maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:48.671136 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.671060 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-maas-api-tls\") pod \"maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:48.671353 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:16:48.671191 5899 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-serving-cert: secret "maas-api-e2e-inf-a-e6f4fe-serving-cert" not found Nov 02 08:16:48.671353 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:16:48.671267 5899 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-maas-api-tls podName:16310be4-c7d0-44de-bfee-579839e26705 nodeName:}" failed. No retries permitted until 2026-11-02 08:16:49.171247733 +0000 UTC m=+3109.469528536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-maas-api-tls") pod "maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" (UID: "16310be4-c7d0-44de-bfee-579839e26705") : secret "maas-api-e2e-inf-a-e6f4fe-serving-cert" not found Nov 02 08:16:48.671874 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.671849 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16310be4-c7d0-44de-bfee-579839e26705-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:48.673503 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.673475 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-metrics-certs\") pod \"maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:48.697873 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.697841 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78dm8\" (UniqueName: \"kubernetes.io/projected/16310be4-c7d0-44de-bfee-579839e26705-kube-api-access-78dm8\") pod \"maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:48.730435 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.730388 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:48.923497 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:48.923401 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:49.178945 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:49.178844 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-maas-api-tls\") pod \"maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:49.180797 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:49.180753 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:49.181662 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:49.181633 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-maas-api-tls\") pod \"maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:49.433039 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:49.432939 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:49.459557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:49.459526 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:16:49.787859 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:49.787595 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs"] Nov 02 08:16:49.790612 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:16:49.790583 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16310be4_c7d0_44de_bfee_579839e26705.slice/crio-f82bc473f5bed94871299453ec6a7110118b2b19dde30d6bb5cf9a20b26aed89 WatchSource:0}: Error finding container f82bc473f5bed94871299453ec6a7110118b2b19dde30d6bb5cf9a20b26aed89: Status 404 returned error can't find the container with id f82bc473f5bed94871299453ec6a7110118b2b19dde30d6bb5cf9a20b26aed89 Nov 02 08:16:50.513721 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:50.513679 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" event={"ID":"16310be4-c7d0-44de-bfee-579839e26705","Type":"ContainerStarted","Data":"83c2c31a7700180f5119c16d7fa98c6f11d3394d11b3f4d8bbea110cb9d1ec98"} Nov 02 08:16:50.514132 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:50.513782 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:50.514132 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:50.513809 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" event={"ID":"16310be4-c7d0-44de-bfee-579839e26705","Type":"ContainerStarted","Data":"f82bc473f5bed94871299453ec6a7110118b2b19dde30d6bb5cf9a20b26aed89"} Nov 02 08:16:50.520788 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:50.520737 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:50.562198 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:50.562110 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" podStartSLOduration=2.2679949649999998 podStartE2EDuration="2.562093435s" totalImagesPullingTime="294.09847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:50.56149565 +0000 UTC m=+3110.859776476" watchObservedRunningTime="2026-11-02 08:16:50.562093435 +0000 UTC m=+3110.860374261" Nov 02 08:16:52.458487 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:52.458454 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:52.608515 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:52.608476 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:52.806317 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:52.806284 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:54.236799 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:54.236758 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:54.389916 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:54.389879 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:54.900482 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:54.900442 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:55.439125 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:55.439088 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:55.566917 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:55.566882 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:56.527090 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:56.527059 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:16:56.828530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:56.828439 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:16:58.814940 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:16:58.814900 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:00.670978 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:00.670937 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:00.821518 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:00.821482 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:02.295816 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.295776 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm"] Nov 02 08:17:02.306256 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.306180 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.327274 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.327242 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm"] Nov 02 08:17:02.334090 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.334056 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e271b1\"/\"default-dockercfg-5nbmc\"" Nov 02 08:17:02.334297 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.334211 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e271b1\"/\"test-model-e271b1-kserve-self-signed-certs\"" Nov 02 08:17:02.336348 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.336327 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e271b1\"/\"openshift-service-ca.crt\"" Nov 02 08:17:02.336504 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.336378 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e271b1\"/\"kube-root-ca.crt\"" Nov 02 08:17:02.388898 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.388844 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:02.470086 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.470039 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-home\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.470319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.470097 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8js6\" (UniqueName: \"kubernetes.io/projected/20d7be3b-a9c6-4a18-ae80-102530c29fdf-kube-api-access-h8js6\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.470394 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.470368 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-dshm\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.470447 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.470417 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tmp-dir\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.470490 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.470482 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tls-certs\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.470546 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.470518 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-model-cache\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.571844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.571730 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-dshm\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.571844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.571797 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tmp-dir\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.571844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.571842 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tls-certs\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.572138 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.571895 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-model-cache\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.572138 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.571988 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-home\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.572138 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.572017 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8js6\" (UniqueName: \"kubernetes.io/projected/20d7be3b-a9c6-4a18-ae80-102530c29fdf-kube-api-access-h8js6\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.573208 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.573177 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tmp-dir\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.573631 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.573607 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-model-cache\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.574013 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.573995 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-home\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.595243 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.576206 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tls-certs\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.601312 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.598937 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-dshm\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.604497 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.604465 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8js6\" (UniqueName: \"kubernetes.io/projected/20d7be3b-a9c6-4a18-ae80-102530c29fdf-kube-api-access-h8js6\") pod \"test-model-e271b1-kserve-56945f9bc5-9kxjm\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.605728 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.605653 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:02.638225 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.638137 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:02.955802 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:02.955777 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm"] Nov 02 08:17:02.957463 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:17:02.957428 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20d7be3b_a9c6_4a18_ae80_102530c29fdf.slice/crio-7210351d41b69afc70543182109767534a161c900fb66d3080c17dbd1f19971e WatchSource:0}: Error finding container 7210351d41b69afc70543182109767534a161c900fb66d3080c17dbd1f19971e: Status 404 returned error can't find the container with id 7210351d41b69afc70543182109767534a161c900fb66d3080c17dbd1f19971e Nov 02 08:17:03.623699 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:03.623662 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" event={"ID":"20d7be3b-a9c6-4a18-ae80-102530c29fdf","Type":"ContainerStarted","Data":"1dea19eacf76539363988f425152c73c68420e1109bbe99ebcb5673da1a925ab"} Nov 02 08:17:03.623699 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:03.623709 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" event={"ID":"20d7be3b-a9c6-4a18-ae80-102530c29fdf","Type":"ContainerStarted","Data":"7210351d41b69afc70543182109767534a161c900fb66d3080c17dbd1f19971e"} Nov 02 08:17:03.639336 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:03.639254 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:03.668099 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:03.668035 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" podStartSLOduration=1.6680186080000001 podStartE2EDuration="1.668018608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:03.666657285 +0000 UTC m=+3123.964938111" watchObservedRunningTime="2026-11-02 08:17:03.668018608 +0000 UTC m=+3123.966299434" Nov 02 08:17:04.274011 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:04.273958 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:06.018487 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:06.018415 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:06.648860 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:06.648827 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:07.305191 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:07.305127 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:07.716320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:07.716210 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:09.480285 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:09.480249 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:11.179914 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:11.179876 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:12.639185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:12.639117 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:12.651844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:12.651807 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:12.702775 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:12.702741 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:17:13.065701 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:13.065661 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:13.143323 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:17:13.143287 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:17:14.827119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:14.827082 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:15.736533 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:15.736496 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:16.571473 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:16.571441 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:18.354940 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:18.354910 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:20.020729 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:20.020687 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:21.733209 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:21.733174 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:23.321577 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.321541 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2"] Nov 02 08:17:23.328373 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.328344 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.344489 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.344459 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2"] Nov 02 08:17:23.346559 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.346532 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f765b8\"/\"kube-root-ca.crt\"" Nov 02 08:17:23.353984 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.353914 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f765b8\"/\"openshift-service-ca.crt\"" Nov 02 08:17:23.356904 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.356877 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f765b8\"/\"default-dockercfg-xghdk\"" Nov 02 08:17:23.357059 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.356995 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f765b8\"/\"test-model-f765b8-kserve-self-signed-certs\"" Nov 02 08:17:23.389783 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.389750 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:23.462453 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.462414 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk2zk\" (UniqueName: \"kubernetes.io/projected/4f680056-7730-47cc-95f4-1493a726d645-kube-api-access-sk2zk\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.462788 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.462731 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-tmp-dir\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.462898 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.462819 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-dshm\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.463002 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.462976 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f680056-7730-47cc-95f4-1493a726d645-tls-certs\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.463098 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.463077 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-home\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.463271 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.463114 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-model-cache\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.537466 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.537428 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:23.564943 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.564903 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-dshm\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.565155 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.565016 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f680056-7730-47cc-95f4-1493a726d645-tls-certs\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.565155 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.565071 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-home\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.565155 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.565099 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-model-cache\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.565377 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.565283 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk2zk\" (UniqueName: \"kubernetes.io/projected/4f680056-7730-47cc-95f4-1493a726d645-kube-api-access-sk2zk\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.565439 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.565373 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-tmp-dir\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.565927 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.565896 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-tmp-dir\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.566559 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.566534 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-home\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.566809 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.566784 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-model-cache\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.575239 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.574014 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f680056-7730-47cc-95f4-1493a726d645-tls-certs\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.584891 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.584817 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-dshm\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.596384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.596333 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk2zk\" (UniqueName: \"kubernetes.io/projected/4f680056-7730-47cc-95f4-1493a726d645-kube-api-access-sk2zk\") pod \"test-model-f765b8-kserve-6c97ccf9f9-qngv2\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.644397 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.644350 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:23.959854 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:23.959822 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2"] Nov 02 08:17:23.960977 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:17:23.960946 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f680056_7730_47cc_95f4_1493a726d645.slice/crio-1dde26a83fbde3748fa1bbd22978783075a77b7b20ce6a52977d150b15625ade WatchSource:0}: Error finding container 1dde26a83fbde3748fa1bbd22978783075a77b7b20ce6a52977d150b15625ade: Status 404 returned error can't find the container with id 1dde26a83fbde3748fa1bbd22978783075a77b7b20ce6a52977d150b15625ade Nov 02 08:17:24.785132 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:24.785090 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" event={"ID":"4f680056-7730-47cc-95f4-1493a726d645","Type":"ContainerStarted","Data":"55f1ca60a2adc06194b7546590969d0d46a5ff457ee2e660290e3e7f284b7cb8"} Nov 02 08:17:24.785132 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:24.785138 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" event={"ID":"4f680056-7730-47cc-95f4-1493a726d645","Type":"ContainerStarted","Data":"1dde26a83fbde3748fa1bbd22978783075a77b7b20ce6a52977d150b15625ade"} Nov 02 08:17:24.830610 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:24.830554 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" podStartSLOduration=1.8305376230000001 podStartE2EDuration="1.830537623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:24.827130374 +0000 UTC m=+3145.125411199" watchObservedRunningTime="2026-11-02 08:17:24.830537623 +0000 UTC m=+3145.128818448" Nov 02 08:17:25.224767 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:25.224679 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:25.645011 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:25.644966 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:26.998964 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:26.998928 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:28.679823 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:28.679791 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:28.816997 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:28.816962 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:28.954353 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:28.954272 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:30.614449 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:30.614410 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:32.410424 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:32.410391 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:33.644807 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:33.644775 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:33.658539 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:33.658509 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:33.866119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:33.866089 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:17:34.199906 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:34.199871 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:36.001785 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:36.001750 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:36.142792 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:36.142756 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:37.830929 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:37.830895 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:39.048429 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:39.048388 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:39.614763 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:39.614726 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:41.316539 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.315175 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr"] Nov 02 08:17:41.321204 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.321125 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.351942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.351911 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c4489d\"/\"default-dockercfg-vnmk8\"" Nov 02 08:17:41.352122 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.351969 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c4489d\"/\"test-model-c4489d-kserve-self-signed-certs\"" Nov 02 08:17:41.352122 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.352031 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c4489d\"/\"openshift-service-ca.crt\"" Nov 02 08:17:41.352122 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.352056 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c4489d\"/\"kube-root-ca.crt\"" Nov 02 08:17:41.355507 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.355482 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr"] Nov 02 08:17:41.396220 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.396185 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:41.433946 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.433891 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9757\" (UniqueName: \"kubernetes.io/projected/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-kube-api-access-s9757\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.434207 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.434062 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-model-cache\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.434309 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.434244 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-home\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.434420 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.434392 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-dshm\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.434551 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.434477 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tmp-dir\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.434551 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.434515 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tls-certs\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.535375 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.535340 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-dshm\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.535580 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.535385 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tmp-dir\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.535580 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.535417 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tls-certs\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.535580 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.535470 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9757\" (UniqueName: \"kubernetes.io/projected/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-kube-api-access-s9757\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.535580 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.535575 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-model-cache\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.535827 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.535671 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-home\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.535883 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.535868 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tmp-dir\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.536103 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.536079 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-model-cache\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.536211 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.536083 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-home\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.536532 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.536415 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:41.537949 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.537919 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-dshm\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.538771 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.538745 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tls-certs\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.570227 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.561031 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9757\" (UniqueName: \"kubernetes.io/projected/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-kube-api-access-s9757\") pod \"test-model-c4489d-kserve-5fb864c89f-dvzbr\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.644332 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.643858 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:41.963811 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:41.963784 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr"] Nov 02 08:17:41.965319 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:17:41.965287 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97ce128f_f1ec_4ecf_904d_ed9481ff6e0b.slice/crio-4be723b4d51effdbba5f4a8570a04f3e0fb8a4d4f49575a0e447faa1bddff159 WatchSource:0}: Error finding container 4be723b4d51effdbba5f4a8570a04f3e0fb8a4d4f49575a0e447faa1bddff159: Status 404 returned error can't find the container with id 4be723b4d51effdbba5f4a8570a04f3e0fb8a4d4f49575a0e447faa1bddff159 Nov 02 08:17:42.925052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:42.925015 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" event={"ID":"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b","Type":"ContainerStarted","Data":"9d2ec49b01307d68c043524555155c548e56d14d380ed46f7a1e8e5a7f0c35c8"} Nov 02 08:17:42.925474 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:42.925062 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" event={"ID":"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b","Type":"ContainerStarted","Data":"4be723b4d51effdbba5f4a8570a04f3e0fb8a4d4f49575a0e447faa1bddff159"} Nov 02 08:17:42.971195 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:42.971111 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" podStartSLOduration=1.971098745 podStartE2EDuration="1.971098745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:42.967386608 +0000 UTC m=+3163.265667433" watchObservedRunningTime="2026-11-02 08:17:42.971098745 +0000 UTC m=+3163.269379602" Nov 02 08:17:43.143181 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:17:43.143121 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:17:43.350891 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:43.350857 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:43.644038 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:43.643958 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:45.026631 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:45.026597 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:47.183763 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:47.183720 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:47.394873 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:47.394838 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:49.198717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:49.198685 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:50.158219 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:50.158150 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:50.594240 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:50.594155 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:51.090022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:51.089987 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:51.269717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:51.269656 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:51.644006 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:51.643959 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:51.656707 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:51.656678 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:52.005956 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:52.005926 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:17:53.008381 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:53.008348 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:54.594841 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:54.594798 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:54.742608 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:54.742572 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:54.901683 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:54.901549 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:56.376391 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:56.366045 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:56.513699 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:56.513666 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:56.703961 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:56.703845 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:17:58.577971 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:17:58.577932 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:00.433342 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:00.433310 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:02.301358 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:02.301313 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:03.660077 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.658850 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq"] Nov 02 08:18:03.663683 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.663656 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.683697 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.683665 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq"] Nov 02 08:18:03.692490 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.692460 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dfd6f0\"/\"kube-root-ca.crt\"" Nov 02 08:18:03.693105 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.693084 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dfd6f0\"/\"openshift-service-ca.crt\"" Nov 02 08:18:03.693248 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.693233 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dfd6f0\"/\"default-dockercfg-frjqw\"" Nov 02 08:18:03.693333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.693320 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dfd6f0\"/\"test-model-dfd6f0-kserve-self-signed-certs\"" Nov 02 08:18:03.735408 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.735369 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:03.762629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.762593 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-model-cache\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.762838 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.762730 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-home\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.762838 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.762784 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tmp-dir\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.762838 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.762824 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-dshm\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.762966 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.762933 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtrn2\" (UniqueName: \"kubernetes.io/projected/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-kube-api-access-rtrn2\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.763015 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.762981 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tls-certs\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.864287 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.864249 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-home\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.864475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.864300 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tmp-dir\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.864475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.864340 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-dshm\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.864475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.864435 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtrn2\" (UniqueName: \"kubernetes.io/projected/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-kube-api-access-rtrn2\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.864587 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.864496 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tls-certs\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.864622 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.864592 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-model-cache\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.864906 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.864756 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-home\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.864906 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.864844 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tmp-dir\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.865131 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.864935 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-model-cache\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.867193 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.867135 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-dshm\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.867493 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.867469 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tls-certs\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.896143 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.892113 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtrn2\" (UniqueName: \"kubernetes.io/projected/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-kube-api-access-rtrn2\") pod \"test-model-dfd6f0-kserve-759f6d9c5-xppxq\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:03.979982 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:03.979954 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:04.274806 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:04.274728 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:04.305258 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:04.305214 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq"] Nov 02 08:18:04.307878 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:18:04.307845 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c2f5b4f_7f8e_426c_a9e7_92aca079659d.slice/crio-6ed862d17d19ccae293c1e5b349d84961add92732eee7623d7f2a48c1bd9bf5e WatchSource:0}: Error finding container 6ed862d17d19ccae293c1e5b349d84961add92732eee7623d7f2a48c1bd9bf5e: Status 404 returned error can't find the container with id 6ed862d17d19ccae293c1e5b349d84961add92732eee7623d7f2a48c1bd9bf5e Nov 02 08:18:05.086040 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:05.086004 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" event={"ID":"4c2f5b4f-7f8e-426c-a9e7-92aca079659d","Type":"ContainerStarted","Data":"22c36d9d9c6edddfbf02e22186246d3a7ff4343c086e85652c5b1f36b8131b4d"} Nov 02 08:18:05.086486 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:05.086055 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" event={"ID":"4c2f5b4f-7f8e-426c-a9e7-92aca079659d","Type":"ContainerStarted","Data":"6ed862d17d19ccae293c1e5b349d84961add92732eee7623d7f2a48c1bd9bf5e"} Nov 02 08:18:05.132703 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:05.132648 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" podStartSLOduration=2.132632597 podStartE2EDuration="2.132632597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:05.129397148 +0000 UTC m=+3185.427677973" watchObservedRunningTime="2026-11-02 08:18:05.132632597 +0000 UTC m=+3185.430913466" Nov 02 08:18:05.980800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:05.980766 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:06.063747 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:06.063715 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:07.424723 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:07.424688 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:07.834208 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:07.834156 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:11.181129 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:11.181090 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:11.513008 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:11.512974 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:13.142796 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:18:13.142748 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:18:13.306797 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:13.306757 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:13.980336 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:13.980302 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:13.992632 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:13.992602 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:14.163682 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:14.163627 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:18:15.138714 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:15.138678 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:17.043881 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:17.043841 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:18.785986 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:18.785945 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:20.609022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:20.608984 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:22.399458 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:22.399419 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:24.159359 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:24.159322 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:25.913268 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:25.913235 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:27.731614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:27.731530 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:29.599473 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:29.599432 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:31.265362 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:31.265311 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:32.731043 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:32.730999 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:33.150703 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:33.150619 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:33.291333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:33.291297 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:33.676634 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:33.676600 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:34.124127 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:34.124085 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:35.019030 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:35.018991 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:36.937913 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:36.937875 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:37.581264 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:37.580817 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:37.737270 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:37.737228 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:38.859206 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:38.859140 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:39.345658 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:39.345624 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:39.492350 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:39.492317 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:40.898742 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:40.898697 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:42.936445 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:42.936407 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:43.141413 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:18:43.141374 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:18:44.825594 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:44.825556 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:46.873616 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:46.873565 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:49.152363 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:49.152308 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:50.121216 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:50.121133 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:50.526522 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:50.526487 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:51.034584 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:51.034546 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:51.279498 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:51.279459 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:52.999453 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:52.999421 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9"] Nov 02 08:18:53.004369 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.004350 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.035155 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.035118 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ba44ed-serving-cert\"" Nov 02 08:18:53.039245 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.039220 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9"] Nov 02 08:18:53.148971 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.148922 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.149171 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.149040 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-metrics-certs\") pod \"maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.149276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.149254 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71135335-1f07-4594-b522-1a79379b31b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.149320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.149303 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b5bv\" (UniqueName: \"kubernetes.io/projected/71135335-1f07-4594-b522-1a79379b31b6-kube-api-access-9b5bv\") pod \"maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.251211 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.251059 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71135335-1f07-4594-b522-1a79379b31b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.251211 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.251151 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b5bv\" (UniqueName: \"kubernetes.io/projected/71135335-1f07-4594-b522-1a79379b31b6-kube-api-access-9b5bv\") pod \"maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.252826 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.251272 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.252826 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.251370 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-metrics-certs\") pod \"maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.252826 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.252306 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71135335-1f07-4594-b522-1a79379b31b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.257026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.255925 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.257719 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.257691 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-metrics-certs\") pod \"maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.265644 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.265598 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:53.276940 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.276902 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b5bv\" (UniqueName: \"kubernetes.io/projected/71135335-1f07-4594-b522-1a79379b31b6-kube-api-access-9b5bv\") pod \"maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.320867 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.320830 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:53.636031 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:53.635991 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9"] Nov 02 08:18:53.637490 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:18:53.637460 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71135335_1f07_4594_b522_1a79379b31b6.slice/crio-83362aac7206f2c288c04bb298e6eea87166cbdb52568b15d912ea4c61078ca8 WatchSource:0}: Error finding container 83362aac7206f2c288c04bb298e6eea87166cbdb52568b15d912ea4c61078ca8: Status 404 returned error can't find the container with id 83362aac7206f2c288c04bb298e6eea87166cbdb52568b15d912ea4c61078ca8 Nov 02 08:18:54.435810 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:54.435729 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" event={"ID":"71135335-1f07-4594-b522-1a79379b31b6","Type":"ContainerStarted","Data":"841f120ab226b31567f4b5d34bd7c3c71c004a927518491f26c5355d951b94ed"} Nov 02 08:18:54.435810 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:54.435783 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" event={"ID":"71135335-1f07-4594-b522-1a79379b31b6","Type":"ContainerStarted","Data":"83362aac7206f2c288c04bb298e6eea87166cbdb52568b15d912ea4c61078ca8"} Nov 02 08:18:54.436276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:54.435951 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:18:54.495391 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:54.495340 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" podStartSLOduration=2.138690093 podStartE2EDuration="2.495323856s" totalImagesPullingTime="356.633763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:54.495263127 +0000 UTC m=+3234.793543954" watchObservedRunningTime="2026-11-02 08:18:54.495323856 +0000 UTC m=+3234.793604701" Nov 02 08:18:54.961715 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:54.961683 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:55.182410 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:55.178497 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:55.788718 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:55.788681 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:56.778342 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:56.778304 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:56.945251 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:56.945212 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:18:58.091515 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:18:58.091479 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:00.412820 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:00.412780 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:00.449776 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:00.449744 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:19:02.395455 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:02.395413 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:04.327207 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:04.327174 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:06.294271 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:06.294238 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:07.050842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:07.050810 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:07.552482 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:07.552443 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:08.087459 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:08.087422 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:10.216708 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:10.216675 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:12.197837 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:12.197716 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:13.143099 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:19:13.143060 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:19:14.085384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:14.085347 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:16.031304 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:16.031273 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:17.811412 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:17.811377 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:19.620848 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:19.620803 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:21.453225 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:21.453155 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:23.319632 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:23.319599 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:25.180532 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:25.180495 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:25.237218 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:25.237154 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq"] Nov 02 08:19:25.237485 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:25.237451 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" podUID="880640a1-42c9-4be2-8b81-c7af629b8a50" containerName="istio-proxy" containerID="cri-o://725b7097307d9fac98c6aabf2ccb6befed2882ce6f0b0fdef3bdba829faf0f44" gracePeriod=30 Nov 02 08:19:27.026100 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:27.026057 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:28.850300 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:28.850265 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:30.506734 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.506656 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:30.511471 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.511445 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:19:30.673916 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.673882 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-data\") pod \"880640a1-42c9-4be2-8b81-c7af629b8a50\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " Nov 02 08:19:30.674113 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.673932 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-kube-api-access-blnxk\" (UniqueName: \"kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-kube-api-access-blnxk\") pod \"880640a1-42c9-4be2-8b81-c7af629b8a50\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " Nov 02 08:19:30.674113 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.673954 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-token\" (UniqueName: \"kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-token\") pod \"880640a1-42c9-4be2-8b81-c7af629b8a50\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " Nov 02 08:19:30.674113 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.674064 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-certs\") pod \"880640a1-42c9-4be2-8b81-c7af629b8a50\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " Nov 02 08:19:30.674331 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.674244 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880640a1-42c9-4be2-8b81-c7af629b8a50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880640a1-42c9-4be2-8b81-c7af629b8a50-istiod-ca-cert\") pod \"880640a1-42c9-4be2-8b81-c7af629b8a50\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " Nov 02 08:19:30.674395 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.674341 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-podinfo\") pod \"880640a1-42c9-4be2-8b81-c7af629b8a50\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " Nov 02 08:19:30.674395 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.674240 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-data" pod "880640a1-42c9-4be2-8b81-c7af629b8a50" (UID: "880640a1-42c9-4be2-8b81-c7af629b8a50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:30.674395 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.674383 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-credential-socket\") pod \"880640a1-42c9-4be2-8b81-c7af629b8a50\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " Nov 02 08:19:30.674572 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.674413 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-socket\") pod \"880640a1-42c9-4be2-8b81-c7af629b8a50\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " Nov 02 08:19:30.674572 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.674409 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-certs" pod "880640a1-42c9-4be2-8b81-c7af629b8a50" (UID: "880640a1-42c9-4be2-8b81-c7af629b8a50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:30.674572 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.674484 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-envoy\") pod \"880640a1-42c9-4be2-8b81-c7af629b8a50\" (UID: \"880640a1-42c9-4be2-8b81-c7af629b8a50\") " Nov 02 08:19:30.674745 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.674613 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-credential-socket" pod "880640a1-42c9-4be2-8b81-c7af629b8a50" (UID: "880640a1-42c9-4be2-8b81-c7af629b8a50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:30.674745 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.674705 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-socket" pod "880640a1-42c9-4be2-8b81-c7af629b8a50" (UID: "880640a1-42c9-4be2-8b81-c7af629b8a50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:30.674865 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.674834 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880640a1-42c9-4be2-8b81-c7af629b8a50-istiod-ca-cert" pod "880640a1-42c9-4be2-8b81-c7af629b8a50" (UID: "880640a1-42c9-4be2-8b81-c7af629b8a50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:19:30.675216 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.675194 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:19:30.675384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.675359 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:19:30.675500 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.675387 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:19:30.675500 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.675399 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:19:30.675500 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.675411 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/880640a1-42c9-4be2-8b81-c7af629b8a50-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:19:30.676597 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.676569 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-kube-api-access-blnxk" pod "880640a1-42c9-4be2-8b81-c7af629b8a50" (UID: "880640a1-42c9-4be2-8b81-c7af629b8a50"). InnerVolumeSpecName "kube-api-access-blnxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:19:30.676826 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.676802 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-token" pod "880640a1-42c9-4be2-8b81-c7af629b8a50" (UID: "880640a1-42c9-4be2-8b81-c7af629b8a50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:19:30.676909 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.676825 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-podinfo" pod "880640a1-42c9-4be2-8b81-c7af629b8a50" (UID: "880640a1-42c9-4be2-8b81-c7af629b8a50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:19:30.677134 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.677113 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-envoy" pod "880640a1-42c9-4be2-8b81-c7af629b8a50" (UID: "880640a1-42c9-4be2-8b81-c7af629b8a50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:19:30.680633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.680602 5899 generic.go:356] "Generic (PLEG): container finished" podID="880640a1-42c9-4be2-8b81-c7af629b8a50" containerID="725b7097307d9fac98c6aabf2ccb6befed2882ce6f0b0fdef3bdba829faf0f44" exitCode=0 Nov 02 08:19:30.680727 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.680674 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" Nov 02 08:19:30.680727 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.680688 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" event={"ID":"880640a1-42c9-4be2-8b81-c7af629b8a50","Type":"ContainerDied","Data":"725b7097307d9fac98c6aabf2ccb6befed2882ce6f0b0fdef3bdba829faf0f44"} Nov 02 08:19:30.680819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.680765 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq" event={"ID":"880640a1-42c9-4be2-8b81-c7af629b8a50","Type":"ContainerDied","Data":"598d1624af0153cb0f76a2488a209fb60108ed74f7251673c00c8aa2c012a1cc"} Nov 02 08:19:30.680819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.680800 5899 scope.go:122] "RemoveContainer" containerID="725b7097307d9fac98c6aabf2ccb6befed2882ce6f0b0fdef3bdba829faf0f44" Nov 02 08:19:30.719245 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.719216 5899 scope.go:122] "RemoveContainer" containerID="725b7097307d9fac98c6aabf2ccb6befed2882ce6f0b0fdef3bdba829faf0f44" Nov 02 08:19:30.719559 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:19:30.719536 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725b7097307d9fac98c6aabf2ccb6befed2882ce6f0b0fdef3bdba829faf0f44\": container with ID starting with 725b7097307d9fac98c6aabf2ccb6befed2882ce6f0b0fdef3bdba829faf0f44 not found: ID does not exist" containerID="725b7097307d9fac98c6aabf2ccb6befed2882ce6f0b0fdef3bdba829faf0f44" Nov 02 08:19:30.719659 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.719570 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725b7097307d9fac98c6aabf2ccb6befed2882ce6f0b0fdef3bdba829faf0f44"} err="failed to get container status \"725b7097307d9fac98c6aabf2ccb6befed2882ce6f0b0fdef3bdba829faf0f44\": rpc error: code = NotFound desc = could not find container \"725b7097307d9fac98c6aabf2ccb6befed2882ce6f0b0fdef3bdba829faf0f44\": container with ID starting with 725b7097307d9fac98c6aabf2ccb6befed2882ce6f0b0fdef3bdba829faf0f44 not found: ID does not exist" Nov 02 08:19:30.757892 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.757821 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq"] Nov 02 08:19:30.766155 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.766128 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f6dgq"] Nov 02 08:19:30.776565 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.776539 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:19:30.776565 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.776566 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:19:30.776713 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.776578 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blnxk\" (UniqueName: \"kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-kube-api-access-blnxk\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:19:30.776713 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:30.776587 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/880640a1-42c9-4be2-8b81-c7af629b8a50-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:19:31.820686 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:31.820654 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880640a1-42c9-4be2-8b81-c7af629b8a50" path="/var/lib/kubelet/pods/880640a1-42c9-4be2-8b81-c7af629b8a50/volumes" Nov 02 08:19:32.232341 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:32.232312 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:34.032637 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:34.032601 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:35.851683 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:35.851651 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:37.546839 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:37.546801 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:39.163375 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.163339 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8"] Nov 02 08:19:39.164016 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.164002 5899 cpu_manager.go:414] "containerMap: removing container" podUID="880640a1-42c9-4be2-8b81-c7af629b8a50" containerName="istio-proxy" Nov 02 08:19:39.164072 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.164017 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880640a1-42c9-4be2-8b81-c7af629b8a50" containerName="istio-proxy" Nov 02 08:19:39.164154 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.164145 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="880640a1-42c9-4be2-8b81-c7af629b8a50" containerName="istio-proxy" Nov 02 08:19:39.167963 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.167940 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:39.189537 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.189502 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8"] Nov 02 08:19:39.193350 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.193313 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrc4m\" (UniqueName: \"kubernetes.io/projected/e18ba88c-ba9d-4ab4-be5e-284c239a9228-kube-api-access-mrc4m\") pod \"maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:39.193484 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.193382 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18ba88c-ba9d-4ab4-be5e-284c239a9228-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:39.193542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.193505 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-maas-api-tls\") pod \"maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:39.193596 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.193578 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-metrics-certs\") pod \"maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:39.197384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.197354 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d042a8-serving-cert\"" Nov 02 08:19:39.287655 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.287618 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:39.295196 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.295152 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrc4m\" (UniqueName: \"kubernetes.io/projected/e18ba88c-ba9d-4ab4-be5e-284c239a9228-kube-api-access-mrc4m\") pod \"maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:39.295334 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.295221 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18ba88c-ba9d-4ab4-be5e-284c239a9228-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:39.295334 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.295250 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-maas-api-tls\") pod \"maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:39.295334 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.295279 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-metrics-certs\") pod \"maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:39.295501 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:19:39.295375 5899 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-serving-cert: secret "maas-api-e2e-inf-b-d042a8-serving-cert" not found Nov 02 08:19:39.295501 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:19:39.295453 5899 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-maas-api-tls podName:e18ba88c-ba9d-4ab4-be5e-284c239a9228 nodeName:}" failed. No retries permitted until 2026-11-02 08:19:39.795436984 +0000 UTC m=+3280.093717801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-maas-api-tls") pod "maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" (UID: "e18ba88c-ba9d-4ab4-be5e-284c239a9228") : secret "maas-api-e2e-inf-b-d042a8-serving-cert" not found Nov 02 08:19:39.295988 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.295963 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18ba88c-ba9d-4ab4-be5e-284c239a9228-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:39.297717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.297698 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-metrics-certs\") pod \"maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:39.322752 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.322712 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrc4m\" (UniqueName: \"kubernetes.io/projected/e18ba88c-ba9d-4ab4-be5e-284c239a9228-kube-api-access-mrc4m\") pod \"maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:39.485064 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.485013 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:39.686194 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.686122 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:39.801884 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.801798 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-maas-api-tls\") pod \"maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:39.804141 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:39.804122 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-maas-api-tls\") pod \"maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:40.083069 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:40.082974 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:40.416527 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:40.416501 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8"] Nov 02 08:19:40.417316 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:19:40.417293 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18ba88c_ba9d_4ab4_be5e_284c239a9228.slice/crio-ac97a135bca4ef6e675316cd1b41aa0cd7bc26e14bfa798c560850571f1a2e7d WatchSource:0}: Error finding container ac97a135bca4ef6e675316cd1b41aa0cd7bc26e14bfa798c560850571f1a2e7d: Status 404 returned error can't find the container with id ac97a135bca4ef6e675316cd1b41aa0cd7bc26e14bfa798c560850571f1a2e7d Nov 02 08:19:40.418834 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:40.418813 5899 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:19:40.752039 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:40.751965 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" event={"ID":"e18ba88c-ba9d-4ab4-be5e-284c239a9228","Type":"ContainerStarted","Data":"ac97a135bca4ef6e675316cd1b41aa0cd7bc26e14bfa798c560850571f1a2e7d"} Nov 02 08:19:41.259629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:41.259586 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:41.761404 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:41.761364 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" event={"ID":"e18ba88c-ba9d-4ab4-be5e-284c239a9228","Type":"ContainerStarted","Data":"014b973fedd488a6a1fcfb4b9410bd5f2c5a537fd339e57436d876a20ea048af"} Nov 02 08:19:41.761851 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:41.761426 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:41.812305 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:41.812235 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" podStartSLOduration=2.425061109 podStartE2EDuration="2.812217973s" totalImagesPullingTime="387.156864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:19:41.810824233 +0000 UTC m=+3282.109105059" watchObservedRunningTime="2026-11-02 08:19:41.812217973 +0000 UTC m=+3282.110498801" Nov 02 08:19:43.018056 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:43.018020 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:43.142014 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:19:43.141975 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:19:44.727750 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:44.727717 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:46.571463 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:46.571400 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:47.774368 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:47.774336 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:19:48.284179 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:48.284137 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:50.014094 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:50.014062 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:51.683762 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:51.683727 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:53.361391 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:53.361351 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:55.322824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:55.322787 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:56.981247 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:56.981207 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz"] Nov 02 08:19:56.986573 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:56.986551 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.016310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.016275 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-eb6010-serving-cert\"" Nov 02 08:19:57.021692 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.021662 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz"] Nov 02 08:19:57.052214 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.052176 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1880d50-8522-44ed-bfff-a4567776155c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.052405 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.052251 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln8w7\" (UniqueName: \"kubernetes.io/projected/a1880d50-8522-44ed-bfff-a4567776155c-kube-api-access-ln8w7\") pod \"maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.052405 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.052397 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-metrics-certs\") pod \"maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.052537 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.052426 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.145865 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.145829 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:57.153837 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.153807 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1880d50-8522-44ed-bfff-a4567776155c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.153987 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.153905 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln8w7\" (UniqueName: \"kubernetes.io/projected/a1880d50-8522-44ed-bfff-a4567776155c-kube-api-access-ln8w7\") pod \"maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.154052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.153995 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-metrics-certs\") pod \"maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.154052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.154040 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.154205 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:19:57.154142 5899 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-serving-cert: secret "maas-api-e2e-inf-b-eb6010-serving-cert" not found Nov 02 08:19:57.154283 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:19:57.154235 5899 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-maas-api-tls podName:a1880d50-8522-44ed-bfff-a4567776155c nodeName:}" failed. No retries permitted until 2026-11-02 08:19:57.654214 +0000 UTC m=+3297.952494822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-maas-api-tls") pod "maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" (UID: "a1880d50-8522-44ed-bfff-a4567776155c") : secret "maas-api-e2e-inf-b-eb6010-serving-cert" not found Nov 02 08:19:57.154806 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.154786 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1880d50-8522-44ed-bfff-a4567776155c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.157075 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.157050 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-metrics-certs\") pod \"maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.179600 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.179565 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln8w7\" (UniqueName: \"kubernetes.io/projected/a1880d50-8522-44ed-bfff-a4567776155c-kube-api-access-ln8w7\") pod \"maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.300418 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.300335 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:57.446103 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.446059 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:19:57.665020 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.664939 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.667590 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.667566 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:57.901115 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:57.901079 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:58.258859 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:58.258826 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz"] Nov 02 08:19:58.259958 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:19:58.259923 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1880d50_8522_44ed_bfff_a4567776155c.slice/crio-5a18f81646f775470752bb92d6bef42535b6181ca70ca69a260705bb0c034098 WatchSource:0}: Error finding container 5a18f81646f775470752bb92d6bef42535b6181ca70ca69a260705bb0c034098: Status 404 returned error can't find the container with id 5a18f81646f775470752bb92d6bef42535b6181ca70ca69a260705bb0c034098 Nov 02 08:19:58.881174 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:58.881066 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" event={"ID":"a1880d50-8522-44ed-bfff-a4567776155c","Type":"ContainerStarted","Data":"5a820622339b516eaf601287ccc0f97268870d3bded32ef5b3df6c712948837f"} Nov 02 08:19:58.881174 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:58.881130 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" event={"ID":"a1880d50-8522-44ed-bfff-a4567776155c","Type":"ContainerStarted","Data":"5a18f81646f775470752bb92d6bef42535b6181ca70ca69a260705bb0c034098"} Nov 02 08:19:58.881411 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:58.881287 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:19:58.932189 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:58.930523 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" podStartSLOduration=2.565588259 podStartE2EDuration="2.930504592s" totalImagesPullingTime="364.916333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:19:58.925557935 +0000 UTC m=+3299.223838760" watchObservedRunningTime="2026-11-02 08:19:58.930504592 +0000 UTC m=+3299.228785419" Nov 02 08:19:59.302651 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:19:59.302614 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:01.035240 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:01.035142 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:02.496319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:02.496272 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:02.727614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:02.727577 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:04.113886 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:04.113843 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:04.588479 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:04.588444 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:04.895497 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:04.895418 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:20:06.438944 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:06.438900 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:07.695282 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:07.695244 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:08.223569 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:08.223529 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:09.956464 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:09.956423 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:11.726454 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:11.726424 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:13.144633 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:20:13.144591 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:20:13.425030 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:13.424943 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:15.328382 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:15.328347 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:17.048338 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:17.048293 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:18.863350 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:18.863314 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:20.584674 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:20.584630 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:22.323523 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:22.323488 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:24.050901 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:24.050862 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:26.026688 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:26.026636 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:26.878660 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:26.878624 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:28.133883 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:28.133842 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:29.971569 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:29.971538 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:31.901616 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:31.901575 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:33.587720 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:33.587679 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:35.394890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:35.394856 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:37.214937 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:37.214906 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:39.031827 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:39.031788 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:40.853619 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:40.853533 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:42.806189 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:42.806146 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:43.141899 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:20:43.141811 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:20:44.623033 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:44.623000 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:45.465705 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:45.465672 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:46.471931 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.471891 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:46.479605 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.479574 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk"] Nov 02 08:20:46.484767 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.484741 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.506971 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.506931 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk"] Nov 02 08:20:46.517192 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.517144 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-78ddbc\"/\"openshift-service-ca.crt\"" Nov 02 08:20:46.517458 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.517152 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-78ddbc\"/\"kube-root-ca.crt\"" Nov 02 08:20:46.571804 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.571767 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-model-cache\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.572005 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.571919 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9thf\" (UniqueName: \"kubernetes.io/projected/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-kube-api-access-r9thf\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.572005 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.571953 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tmp-dir\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.572005 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.571984 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-dshm\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.572244 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.572055 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-home\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.572315 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.572243 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tls-certs\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.579047 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.579019 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-78ddbc\"/\"default-dockercfg-tjc89\"" Nov 02 08:20:46.579196 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.579063 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-78ddbc\"/\"test-model-78ddbc-kserve-self-signed-certs\"" Nov 02 08:20:46.644127 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.644089 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:46.673416 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.673373 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tmp-dir\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.673416 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.673416 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-dshm\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.673672 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.673436 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-home\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.673672 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.673573 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tls-certs\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.673796 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.673678 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-model-cache\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.673796 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.673785 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9thf\" (UniqueName: \"kubernetes.io/projected/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-kube-api-access-r9thf\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.673908 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.673889 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-home\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.674152 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.674122 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tmp-dir\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.674152 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.674119 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-model-cache\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.676109 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.676088 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tls-certs\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.676247 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.676149 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-dshm\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.699320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.699282 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9thf\" (UniqueName: \"kubernetes.io/projected/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-kube-api-access-r9thf\") pod \"test-model-78ddbc-kserve-747b586487-bp4bk\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:46.801074 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:46.801038 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:47.137763 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:47.137728 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk"] Nov 02 08:20:47.139543 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:20:47.139517 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod991f7ad0_9e4f_40fa_994e_fefbd20f7c09.slice/crio-8da09d74e9f0bc90711c89391782462af42dad9588257533bdabb7e6d60d9298 WatchSource:0}: Error finding container 8da09d74e9f0bc90711c89391782462af42dad9588257533bdabb7e6d60d9298: Status 404 returned error can't find the container with id 8da09d74e9f0bc90711c89391782462af42dad9588257533bdabb7e6d60d9298 Nov 02 08:20:47.235108 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:47.235072 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" event={"ID":"991f7ad0-9e4f-40fa-994e-fefbd20f7c09","Type":"ContainerStarted","Data":"23c6eec312a770dd4d22e8dccc19d0f0e2fecbacc6c4d205b1f7298a9835843b"} Nov 02 08:20:47.235108 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:47.235116 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" event={"ID":"991f7ad0-9e4f-40fa-994e-fefbd20f7c09","Type":"ContainerStarted","Data":"8da09d74e9f0bc90711c89391782462af42dad9588257533bdabb7e6d60d9298"} Nov 02 08:20:47.282147 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:47.282087 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" podStartSLOduration=1.282072081 podStartE2EDuration="1.282072081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:20:47.280677456 +0000 UTC m=+3347.578958283" watchObservedRunningTime="2026-11-02 08:20:47.282072081 +0000 UTC m=+3347.580352906" Nov 02 08:20:47.802119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:47.802081 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:48.382222 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:48.382183 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:50.123803 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:50.123765 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:50.535789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:50.535743 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:52.049050 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:52.049012 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:53.779384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:53.779335 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:55.519475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:55.519435 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:56.802019 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:56.801961 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:56.814991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:56.814962 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:57.296444 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:57.296390 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:20:57.318494 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:57.318466 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:20:58.982813 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:20:58.982768 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:00.937096 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:00.937055 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:02.671771 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:02.671728 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:02.834513 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:02.834471 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:04.645852 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:04.645812 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:06.415003 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:06.414970 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:07.786054 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:07.786022 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:08.202567 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:08.202456 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:09.920818 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:09.920783 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:11.670960 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:11.670921 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:13.159866 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:21:13.159804 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:21:14.080901 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.080862 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:14.886572 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.886534 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt"] Nov 02 08:21:14.895716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.895689 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:14.911577 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.911547 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt"] Nov 02 08:21:14.920485 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.920456 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b72fa3\"/\"openshift-service-ca.crt\"" Nov 02 08:21:14.920485 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.920461 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b72fa3\"/\"test-model-b72fa3-kserve-self-signed-certs\"" Nov 02 08:21:14.920675 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.920510 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b72fa3\"/\"kube-root-ca.crt\"" Nov 02 08:21:14.924638 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.924617 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b72fa3\"/\"default-dockercfg-btnzc\"" Nov 02 08:21:14.968418 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.968381 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:14.992809 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.992771 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-dshm\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:14.992809 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.992808 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-model-cache\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:14.993091 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.992991 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-home\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:14.993091 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.993078 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-tmp-dir\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:14.993230 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.993208 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e766ca8e-466e-4ec7-aacf-758079b07d65-tls-certs\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:14.993290 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:14.993271 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbvhs\" (UniqueName: \"kubernetes.io/projected/e766ca8e-466e-4ec7-aacf-758079b07d65-kube-api-access-sbvhs\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.094640 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.094599 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-home\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.094843 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.094686 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-tmp-dir\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.094843 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.094811 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e766ca8e-466e-4ec7-aacf-758079b07d65-tls-certs\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.094972 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.094894 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbvhs\" (UniqueName: \"kubernetes.io/projected/e766ca8e-466e-4ec7-aacf-758079b07d65-kube-api-access-sbvhs\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.095035 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.095012 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-dshm\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.095092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.095037 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-model-cache\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.095092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.095055 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-home\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.095092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.095065 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-tmp-dir\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.095698 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.095672 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-model-cache\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.097729 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.097702 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-dshm\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.101832 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.101813 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e766ca8e-466e-4ec7-aacf-758079b07d65-tls-certs\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.121695 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.121665 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbvhs\" (UniqueName: \"kubernetes.io/projected/e766ca8e-466e-4ec7-aacf-758079b07d65-kube-api-access-sbvhs\") pod \"test-model-b72fa3-kserve-66947968fc-2snrt\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.212201 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.211815 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:15.545576 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.545541 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt"] Nov 02 08:21:15.546597 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:21:15.546559 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode766ca8e_466e_4ec7_aacf_758079b07d65.slice/crio-0d4c22cf7ca3281913e611720f21aba5066bac9973b0c54c2a39cb2891e7ebce WatchSource:0}: Error finding container 0d4c22cf7ca3281913e611720f21aba5066bac9973b0c54c2a39cb2891e7ebce: Status 404 returned error can't find the container with id 0d4c22cf7ca3281913e611720f21aba5066bac9973b0c54c2a39cb2891e7ebce Nov 02 08:21:15.866682 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:15.866564 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:16.450021 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:16.449977 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" event={"ID":"e766ca8e-466e-4ec7-aacf-758079b07d65","Type":"ContainerStarted","Data":"7a6d45c4bac3f703fe8f55e8768d8286ce3e2a6a9e40b33509031b82bab68d45"} Nov 02 08:21:16.450021 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:16.450024 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" event={"ID":"e766ca8e-466e-4ec7-aacf-758079b07d65","Type":"ContainerStarted","Data":"0d4c22cf7ca3281913e611720f21aba5066bac9973b0c54c2a39cb2891e7ebce"} Nov 02 08:21:16.492754 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:16.492687 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" podStartSLOduration=2.49266805 podStartE2EDuration="2.49266805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:21:16.492102972 +0000 UTC m=+3376.790383808" watchObservedRunningTime="2026-11-02 08:21:16.49266805 +0000 UTC m=+3376.790948876" Nov 02 08:21:17.211959 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:17.211916 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:17.654890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:17.654853 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:19.418978 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:19.418943 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:21.097906 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:21.097871 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:22.858663 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:22.858620 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:24.613318 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:24.613277 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:25.212181 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:25.212125 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:25.224998 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:25.224963 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:25.526525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:25.526494 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:21:26.366462 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:26.366426 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:28.125061 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:28.125011 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:29.354257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.354222 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9"] Nov 02 08:21:29.361872 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.361842 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.378182 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.378132 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9"] Nov 02 08:21:29.386113 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.386082 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-eb6010\"/\"openshift-service-ca.crt\"" Nov 02 08:21:29.386284 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.386144 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-eb6010\"/\"kube-root-ca.crt\"" Nov 02 08:21:29.391887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.391859 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-eb6010\"/\"default-dockercfg-rgmz4\"" Nov 02 08:21:29.391887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.391859 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-eb6010\"/\"test-model-eb6010-kserve-self-signed-certs\"" Nov 02 08:21:29.399731 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.399702 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-dshm\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.399855 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.399800 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-tmp-dir\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.399974 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.399953 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-model-cache\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.400060 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.400028 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a92d409-9023-4994-9c49-86bed431a47a-tls-certs\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.400246 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.400229 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-home\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.400299 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.400273 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqmrm\" (UniqueName: \"kubernetes.io/projected/1a92d409-9023-4994-9c49-86bed431a47a-kube-api-access-xqmrm\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.476859 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.476820 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:29.501659 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.501613 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqmrm\" (UniqueName: \"kubernetes.io/projected/1a92d409-9023-4994-9c49-86bed431a47a-kube-api-access-xqmrm\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.501814 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.501797 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-dshm\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.501871 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.501844 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-tmp-dir\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.501964 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.501949 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-model-cache\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.502054 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.502036 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a92d409-9023-4994-9c49-86bed431a47a-tls-certs\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.502178 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.502144 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-home\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.502405 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.502379 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-tmp-dir\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.502498 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.502479 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-home\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.502620 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.502599 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-model-cache\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.504461 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.504434 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-dshm\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.504752 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.504730 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a92d409-9023-4994-9c49-86bed431a47a-tls-certs\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.564273 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.564233 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqmrm\" (UniqueName: \"kubernetes.io/projected/1a92d409-9023-4994-9c49-86bed431a47a-kube-api-access-xqmrm\") pod \"test-model-eb6010-kserve-869dc6f78c-82wm9\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.678541 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.678440 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:29.997094 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:29.997065 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9"] Nov 02 08:21:29.998857 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:21:29.998827 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a92d409_9023_4994_9c49_86bed431a47a.slice/crio-04e543611873d6723c77cd75b83790a0a2b7876941e7fc3381e275d43ffd0024 WatchSource:0}: Error finding container 04e543611873d6723c77cd75b83790a0a2b7876941e7fc3381e275d43ffd0024: Status 404 returned error can't find the container with id 04e543611873d6723c77cd75b83790a0a2b7876941e7fc3381e275d43ffd0024 Nov 02 08:21:30.097218 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:30.096852 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:30.561275 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:30.561229 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" event={"ID":"1a92d409-9023-4994-9c49-86bed431a47a","Type":"ContainerStarted","Data":"e1c5546360982d8e5d41892156a02bce2750a007d9ba7b5b114fb491388e91d9"} Nov 02 08:21:30.561275 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:30.561277 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" event={"ID":"1a92d409-9023-4994-9c49-86bed431a47a","Type":"ContainerStarted","Data":"04e543611873d6723c77cd75b83790a0a2b7876941e7fc3381e275d43ffd0024"} Nov 02 08:21:30.605530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:30.605477 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" podStartSLOduration=1.6054601 podStartE2EDuration="1.6054601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:21:30.604770233 +0000 UTC m=+3390.903051058" watchObservedRunningTime="2026-11-02 08:21:30.6054601 +0000 UTC m=+3390.903740926" Nov 02 08:21:30.678527 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:30.678492 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:31.779691 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:31.779643 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:33.614960 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:33.614922 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:35.416102 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:35.416054 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:37.232384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:37.232343 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:39.049259 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:39.049217 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:39.679544 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:39.679495 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:39.692648 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:39.692618 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:40.643254 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:40.643223 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:21:40.916608 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:40.916534 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:42.735831 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:42.735796 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:43.142671 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:21:43.142588 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:21:44.517361 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:44.517310 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:46.554570 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:46.554531 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:48.797299 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:48.797257 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:50.527016 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:50.526972 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:51.590498 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:51.590459 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn"] Nov 02 08:21:51.590923 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:51.590674 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" containerID="cri-o://90c6cd63f7fea360de8d4cb0a340eaaac868281840a1e8e0bcc876e375b9d8ae" gracePeriod=30 Nov 02 08:21:52.344104 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:52.344058 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:53.217654 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:53.217612 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm"] Nov 02 08:21:53.274387 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:53.274354 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:54.526025 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:54.525971 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:56.243139 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.243086 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:56.372860 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.372809 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:56.758118 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.758079 5899 generic.go:356] "Generic (PLEG): container finished" podID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerID="90c6cd63f7fea360de8d4cb0a340eaaac868281840a1e8e0bcc876e375b9d8ae" exitCode=0 Nov 02 08:21:56.758118 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.758110 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" event={"ID":"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8","Type":"ContainerDied","Data":"90c6cd63f7fea360de8d4cb0a340eaaac868281840a1e8e0bcc876e375b9d8ae"} Nov 02 08:21:56.883617 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.883539 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 08:21:56.924744 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.924707 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-envoy\") pod \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " Nov 02 08:21:56.924929 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.924753 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-data\") pod \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " Nov 02 08:21:56.924929 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.924797 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-socket\") pod \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " Nov 02 08:21:56.924929 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.924830 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-certs\") pod \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " Nov 02 08:21:56.924929 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.924906 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-kube-api-access-xw96q\" (UniqueName: \"kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-kube-api-access-xw96q\") pod \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " Nov 02 08:21:56.925148 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.924932 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-credential-socket\") pod \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " Nov 02 08:21:56.925148 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.924969 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-token\") pod \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " Nov 02 08:21:56.925148 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.925049 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istiod-ca-cert\") pod \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " Nov 02 08:21:56.925148 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.925061 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-data" pod "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" (UID: "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:21:56.925404 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.925142 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-socket" pod "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" (UID: "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:21:56.925404 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.925257 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-podinfo\") pod \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\" (UID: \"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8\") " Nov 02 08:21:56.925404 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.925298 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-credential-socket" pod "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" (UID: "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:21:56.925404 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.925322 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-certs" pod "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" (UID: "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:21:56.925628 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.925459 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istiod-ca-cert" pod "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" (UID: "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:21:56.925926 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.925895 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:21:56.926022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.925927 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:21:56.926022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.925942 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:21:56.926022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.925953 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:21:56.926022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.925965 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:21:56.927744 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.927718 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-envoy" pod "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" (UID: "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:21:56.927847 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.927795 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-podinfo" pod "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" (UID: "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:21:56.927897 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.927869 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-token" pod "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" (UID: "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:21:56.928055 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:56.928032 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-kube-api-access-xw96q" pod "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" (UID: "f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8"). InnerVolumeSpecName "kube-api-access-xw96q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:21:57.028897 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:57.028858 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw96q\" (UniqueName: \"kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-kube-api-access-xw96q\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:21:57.028897 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:57.028893 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:21:57.028897 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:57.028907 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:21:57.029149 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:57.028918 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:21:57.766379 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:57.766348 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" Nov 02 08:21:57.766870 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:57.766348 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn" event={"ID":"f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8","Type":"ContainerDied","Data":"4976c6e0c1ef1a198da3613b4846e6f0b0998f752bca7915d63ccc761535c671"} Nov 02 08:21:57.766870 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:57.766532 5899 scope.go:122] "RemoveContainer" containerID="90c6cd63f7fea360de8d4cb0a340eaaac868281840a1e8e0bcc876e375b9d8ae" Nov 02 08:21:57.835292 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:57.835258 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn"] Nov 02 08:21:57.840992 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:57.840961 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-t89kn"] Nov 02 08:21:57.913195 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:57.913148 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:59.702483 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:59.702447 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:21:59.822141 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:21:59.822108 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" path="/var/lib/kubelet/pods/f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8/volumes" Nov 02 08:22:01.418946 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:01.418910 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:03.203219 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:03.203151 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:05.029028 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:05.028959 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:06.856877 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:06.856842 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:07.727883 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:07.727842 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47"] Nov 02 08:22:07.728287 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:07.728105 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerName="istio-proxy" containerID="cri-o://e89ec449e80d850230751a585f5499bec9b99d88433ac0a084209ea51ff9617e" gracePeriod=30 Nov 02 08:22:07.760549 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:07.760470 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:07.921282 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:07.921240 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:08.240050 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.240008 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" podUID="20d7be3b-a9c6-4a18-ae80-102530c29fdf" containerName="main" containerID="cri-o://1dea19eacf76539363988f425152c73c68420e1109bbe99ebcb5673da1a925ab" gracePeriod=45 Nov 02 08:22:08.533677 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.533641 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:22:08.568319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.568283 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:08.691458 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.691424 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-home\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-home\") pod \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " Nov 02 08:22:08.691650 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.691570 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-model-cache\") pod \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " Nov 02 08:22:08.691650 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.691603 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tls-certs\" (UniqueName: \"kubernetes.io/secret/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tls-certs\") pod \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " Nov 02 08:22:08.691756 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.691647 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d7be3b-a9c6-4a18-ae80-102530c29fdf-kube-api-access-h8js6\" (UniqueName: \"kubernetes.io/projected/20d7be3b-a9c6-4a18-ae80-102530c29fdf-kube-api-access-h8js6\") pod \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " Nov 02 08:22:08.691756 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.691698 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tmp-dir\") pod \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " Nov 02 08:22:08.691756 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.691713 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-home" pod "20d7be3b-a9c6-4a18-ae80-102530c29fdf" (UID: "20d7be3b-a9c6-4a18-ae80-102530c29fdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:08.691756 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.691735 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-dshm\") pod \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\" (UID: \"20d7be3b-a9c6-4a18-ae80-102530c29fdf\") " Nov 02 08:22:08.691916 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.691898 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-model-cache" pod "20d7be3b-a9c6-4a18-ae80-102530c29fdf" (UID: "20d7be3b-a9c6-4a18-ae80-102530c29fdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:08.692022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.692001 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tmp-dir" pod "20d7be3b-a9c6-4a18-ae80-102530c29fdf" (UID: "20d7be3b-a9c6-4a18-ae80-102530c29fdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:08.692355 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.692332 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:08.692355 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.692354 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:08.692355 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.692362 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:08.693964 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.693939 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-dshm" pod "20d7be3b-a9c6-4a18-ae80-102530c29fdf" (UID: "20d7be3b-a9c6-4a18-ae80-102530c29fdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:08.694078 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.694037 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tls-certs" pod "20d7be3b-a9c6-4a18-ae80-102530c29fdf" (UID: "20d7be3b-a9c6-4a18-ae80-102530c29fdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:22:08.694118 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.694076 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d7be3b-a9c6-4a18-ae80-102530c29fdf-kube-api-access-h8js6" pod "20d7be3b-a9c6-4a18-ae80-102530c29fdf" (UID: "20d7be3b-a9c6-4a18-ae80-102530c29fdf"). InnerVolumeSpecName "kube-api-access-h8js6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:22:08.793731 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.793660 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20d7be3b-a9c6-4a18-ae80-102530c29fdf-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:08.793731 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.793685 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8js6\" (UniqueName: \"kubernetes.io/projected/20d7be3b-a9c6-4a18-ae80-102530c29fdf-kube-api-access-h8js6\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:08.793731 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.793696 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20d7be3b-a9c6-4a18-ae80-102530c29fdf-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:08.854015 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.853977 5899 generic.go:356] "Generic (PLEG): container finished" podID="20d7be3b-a9c6-4a18-ae80-102530c29fdf" containerID="1dea19eacf76539363988f425152c73c68420e1109bbe99ebcb5673da1a925ab" exitCode=0 Nov 02 08:22:08.854225 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.854020 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" Nov 02 08:22:08.854225 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.854058 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" event={"ID":"20d7be3b-a9c6-4a18-ae80-102530c29fdf","Type":"ContainerDied","Data":"1dea19eacf76539363988f425152c73c68420e1109bbe99ebcb5673da1a925ab"} Nov 02 08:22:08.854225 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.854095 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm" event={"ID":"20d7be3b-a9c6-4a18-ae80-102530c29fdf","Type":"ContainerDied","Data":"7210351d41b69afc70543182109767534a161c900fb66d3080c17dbd1f19971e"} Nov 02 08:22:08.854225 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.854112 5899 scope.go:122] "RemoveContainer" containerID="1dea19eacf76539363988f425152c73c68420e1109bbe99ebcb5673da1a925ab" Nov 02 08:22:08.869023 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.868995 5899 scope.go:122] "RemoveContainer" containerID="1dea19eacf76539363988f425152c73c68420e1109bbe99ebcb5673da1a925ab" Nov 02 08:22:08.869362 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:22:08.869344 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dea19eacf76539363988f425152c73c68420e1109bbe99ebcb5673da1a925ab\": container with ID starting with 1dea19eacf76539363988f425152c73c68420e1109bbe99ebcb5673da1a925ab not found: ID does not exist" containerID="1dea19eacf76539363988f425152c73c68420e1109bbe99ebcb5673da1a925ab" Nov 02 08:22:08.869408 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.869373 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dea19eacf76539363988f425152c73c68420e1109bbe99ebcb5673da1a925ab"} err="failed to get container status \"1dea19eacf76539363988f425152c73c68420e1109bbe99ebcb5673da1a925ab\": rpc error: code = NotFound desc = could not find container \"1dea19eacf76539363988f425152c73c68420e1109bbe99ebcb5673da1a925ab\": container with ID starting with 1dea19eacf76539363988f425152c73c68420e1109bbe99ebcb5673da1a925ab not found: ID does not exist" Nov 02 08:22:08.909543 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.909511 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm"] Nov 02 08:22:08.922353 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:08.922319 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e271b1/test-model-e271b1-kserve-56945f9bc5-9kxjm"] Nov 02 08:22:09.823246 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:09.823210 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d7be3b-a9c6-4a18-ae80-102530c29fdf" path="/var/lib/kubelet/pods/20d7be3b-a9c6-4a18-ae80-102530c29fdf/volumes" Nov 02 08:22:10.444333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:10.444294 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:10.702549 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:10.702424 5899 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:22:10.702736 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:10.702564 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:22:12.405137 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:12.405104 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:12.887216 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:12.887180 5899 generic.go:356] "Generic (PLEG): container finished" podID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerID="e89ec449e80d850230751a585f5499bec9b99d88433ac0a084209ea51ff9617e" exitCode=0 Nov 02 08:22:12.887419 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:12.887221 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" event={"ID":"e5cde1bc-a874-4ec0-b40c-c294de1a54bf","Type":"ContainerDied","Data":"e89ec449e80d850230751a585f5499bec9b99d88433ac0a084209ea51ff9617e"} Nov 02 08:22:13.025667 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.025634 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 08:22:13.144717 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:22:13.144662 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:22:13.153895 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.153860 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istiod-ca-cert\") pod \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " Nov 02 08:22:13.154086 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.153913 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-podinfo\") pod \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " Nov 02 08:22:13.154086 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.153992 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-kube-api-access-6kzxh\" (UniqueName: \"kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-kube-api-access-6kzxh\") pod \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " Nov 02 08:22:13.154086 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.154028 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-envoy\") pod \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " Nov 02 08:22:13.154310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.154175 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-certs\") pod \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " Nov 02 08:22:13.154310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.154211 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-socket\") pod \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " Nov 02 08:22:13.154310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.154287 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istiod-ca-cert" pod "e5cde1bc-a874-4ec0-b40c-c294de1a54bf" (UID: "e5cde1bc-a874-4ec0-b40c-c294de1a54bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:22:13.154489 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.154458 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-certs" pod "e5cde1bc-a874-4ec0-b40c-c294de1a54bf" (UID: "e5cde1bc-a874-4ec0-b40c-c294de1a54bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:13.154609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.154467 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-token\" (UniqueName: \"kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-token\") pod \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " Nov 02 08:22:13.154609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.154552 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-socket" pod "e5cde1bc-a874-4ec0-b40c-c294de1a54bf" (UID: "e5cde1bc-a874-4ec0-b40c-c294de1a54bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:13.154609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.154591 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-credential-socket\") pod \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " Nov 02 08:22:13.154793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.154632 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-data\") pod \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\" (UID: \"e5cde1bc-a874-4ec0-b40c-c294de1a54bf\") " Nov 02 08:22:13.154891 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.154871 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-credential-socket" pod "e5cde1bc-a874-4ec0-b40c-c294de1a54bf" (UID: "e5cde1bc-a874-4ec0-b40c-c294de1a54bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:13.155113 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.155091 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-data" pod "e5cde1bc-a874-4ec0-b40c-c294de1a54bf" (UID: "e5cde1bc-a874-4ec0-b40c-c294de1a54bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:13.155435 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.155415 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:13.155488 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.155443 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:13.155488 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.155456 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:13.155488 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.155469 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:13.155488 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.155484 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:13.156565 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.156537 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-podinfo" pod "e5cde1bc-a874-4ec0-b40c-c294de1a54bf" (UID: "e5cde1bc-a874-4ec0-b40c-c294de1a54bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:22:13.156722 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.156695 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-kube-api-access-6kzxh" pod "e5cde1bc-a874-4ec0-b40c-c294de1a54bf" (UID: "e5cde1bc-a874-4ec0-b40c-c294de1a54bf"). InnerVolumeSpecName "kube-api-access-6kzxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:22:13.156947 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.156919 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-envoy" pod "e5cde1bc-a874-4ec0-b40c-c294de1a54bf" (UID: "e5cde1bc-a874-4ec0-b40c-c294de1a54bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:13.156947 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.156932 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-token" pod "e5cde1bc-a874-4ec0-b40c-c294de1a54bf" (UID: "e5cde1bc-a874-4ec0-b40c-c294de1a54bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:22:13.257133 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.257047 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:13.257133 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.257079 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:13.257133 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.257088 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzxh\" (UniqueName: \"kubernetes.io/projected/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-kube-api-access-6kzxh\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:13.257133 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.257097 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cde1bc-a874-4ec0-b40c-c294de1a54bf-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:13.896681 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.896644 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" event={"ID":"e5cde1bc-a874-4ec0-b40c-c294de1a54bf","Type":"ContainerDied","Data":"65f129325815196dfc2ea41e82079e6f7d21b7a3811deb78c1d5b410c62fcb70"} Nov 02 08:22:13.897203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.896718 5899 scope.go:122] "RemoveContainer" containerID="e89ec449e80d850230751a585f5499bec9b99d88433ac0a084209ea51ff9617e" Nov 02 08:22:13.897203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.896659 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47" Nov 02 08:22:13.965802 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.965763 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47"] Nov 02 08:22:13.978856 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:13.978822 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-xvt47"] Nov 02 08:22:14.239324 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:14.239285 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:15.821713 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:15.821675 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" path="/var/lib/kubelet/pods/e5cde1bc-a874-4ec0-b40c-c294de1a54bf/volumes" Nov 02 08:22:16.101831 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:16.101736 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:17.172988 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:17.172942 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2"] Nov 02 08:22:17.246722 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:17.246643 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:17.673341 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:17.673294 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:17.937849 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:17.937768 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:19.489570 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:19.489535 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92"] Nov 02 08:22:19.490038 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:19.489758 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerName="istio-proxy" containerID="cri-o://5a9a00d25b69a299c05c9837d2ee70f8ef3c8a72b6fdba3bdceb84627fc13496" gracePeriod=30 Nov 02 08:22:19.696589 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:19.696556 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:21.409743 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:21.409704 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:21.491837 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:21.491794 5899 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:22:21.492031 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:21.491864 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:22:23.130948 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:23.130897 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:23.835089 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:23.835049 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:23.971024 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:23.970981 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:24.111863 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.111773 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:24.164849 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.164814 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr"] Nov 02 08:22:24.259733 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.259694 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:24.773709 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.773640 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 08:22:24.807292 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.807256 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-socket\") pod \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " Nov 02 08:22:24.807476 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.807389 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-podinfo\") pod \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " Nov 02 08:22:24.807476 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.807415 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-certs\") pod \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " Nov 02 08:22:24.807476 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.807450 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-kube-api-access-qf4gg\" (UniqueName: \"kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-kube-api-access-qf4gg\") pod \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " Nov 02 08:22:24.807645 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.807478 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-envoy\") pod \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " Nov 02 08:22:24.807645 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.807512 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-credential-socket\") pod \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " Nov 02 08:22:24.807645 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.807570 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-socket" pod "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" (UID: "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:24.807645 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.807626 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-token\" (UniqueName: \"kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-token\") pod \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " Nov 02 08:22:24.807858 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.807727 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istiod-ca-cert\") pod \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " Nov 02 08:22:24.807858 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.807789 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-data\") pod \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\" (UID: \"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9\") " Nov 02 08:22:24.808106 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.807788 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-certs" pod "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" (UID: "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:24.808106 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.807983 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-data" pod "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" (UID: "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:24.808283 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.808049 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istiod-ca-cert" pod "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" (UID: "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:22:24.808283 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.808068 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-credential-socket" pod "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" (UID: "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:24.808580 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.808555 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:24.808663 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.808586 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:24.808663 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.808600 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:24.808663 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.808612 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:24.808663 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.808626 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:24.810388 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.810204 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-token" pod "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" (UID: "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:22:24.810388 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.810302 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-kube-api-access-qf4gg" pod "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" (UID: "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9"). InnerVolumeSpecName "kube-api-access-qf4gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:22:24.810388 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.810321 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-envoy" pod "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" (UID: "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:24.810388 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.810365 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-podinfo" pod "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" (UID: "e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:22:24.910132 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.910098 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:24.910132 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.910127 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:24.910132 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.910137 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf4gg\" (UniqueName: \"kubernetes.io/projected/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-kube-api-access-qf4gg\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:24.910132 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.910148 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:24.985953 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.985916 5899 generic.go:356] "Generic (PLEG): container finished" podID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerID="5a9a00d25b69a299c05c9837d2ee70f8ef3c8a72b6fdba3bdceb84627fc13496" exitCode=0 Nov 02 08:22:24.986116 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.985986 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" Nov 02 08:22:24.986116 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.986002 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" event={"ID":"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9","Type":"ContainerDied","Data":"5a9a00d25b69a299c05c9837d2ee70f8ef3c8a72b6fdba3bdceb84627fc13496"} Nov 02 08:22:24.986116 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.986086 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92" event={"ID":"e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9","Type":"ContainerDied","Data":"5959a8a813b085cd7c89363d020bf738d45038bb8e9c08e7bfff8c23c42c1592"} Nov 02 08:22:24.986116 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.986100 5899 scope.go:122] "RemoveContainer" containerID="5a9a00d25b69a299c05c9837d2ee70f8ef3c8a72b6fdba3bdceb84627fc13496" Nov 02 08:22:24.998035 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.998009 5899 scope.go:122] "RemoveContainer" containerID="5a9a00d25b69a299c05c9837d2ee70f8ef3c8a72b6fdba3bdceb84627fc13496" Nov 02 08:22:24.998402 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:22:24.998375 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9a00d25b69a299c05c9837d2ee70f8ef3c8a72b6fdba3bdceb84627fc13496\": container with ID starting with 5a9a00d25b69a299c05c9837d2ee70f8ef3c8a72b6fdba3bdceb84627fc13496 not found: ID does not exist" containerID="5a9a00d25b69a299c05c9837d2ee70f8ef3c8a72b6fdba3bdceb84627fc13496" Nov 02 08:22:24.998468 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:24.998417 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9a00d25b69a299c05c9837d2ee70f8ef3c8a72b6fdba3bdceb84627fc13496"} err="failed to get container status \"5a9a00d25b69a299c05c9837d2ee70f8ef3c8a72b6fdba3bdceb84627fc13496\": rpc error: code = NotFound desc = could not find container \"5a9a00d25b69a299c05c9837d2ee70f8ef3c8a72b6fdba3bdceb84627fc13496\": container with ID starting with 5a9a00d25b69a299c05c9837d2ee70f8ef3c8a72b6fdba3bdceb84627fc13496 not found: ID does not exist" Nov 02 08:22:25.041973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:25.041888 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92"] Nov 02 08:22:25.048657 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:25.048624 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-rjj92"] Nov 02 08:22:25.140708 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:25.140676 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:25.821434 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:25.821400 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" path="/var/lib/kubelet/pods/e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9/volumes" Nov 02 08:22:26.857804 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:26.857769 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:27.758098 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:27.758062 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq"] Nov 02 08:22:27.758355 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:27.758330 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" podUID="57768ba3-8718-4280-8205-1827026315b9" containerName="istio-proxy" containerID="cri-o://8c9694f65a210fc374a040ee254de633b63ba60cb8360e0d7df37b5a9d5e7b76" gracePeriod=30 Nov 02 08:22:28.602951 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:28.602898 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:30.362923 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:30.362884 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:30.504741 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:30.504707 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:32.205110 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.205072 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" podUID="4f680056-7730-47cc-95f4-1493a726d645" containerName="main" containerID="cri-o://55f1ca60a2adc06194b7546590969d0d46a5ff457ee2e660290e3e7f284b7cb8" gracePeriod=45 Nov 02 08:22:32.270455 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.270418 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:32.494806 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.494779 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:22:32.622026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.621984 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-tmp-dir\") pod \"4f680056-7730-47cc-95f4-1493a726d645\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " Nov 02 08:22:32.622251 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.622035 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-model-cache\") pod \"4f680056-7730-47cc-95f4-1493a726d645\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " Nov 02 08:22:32.622251 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.622108 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f680056-7730-47cc-95f4-1493a726d645-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f680056-7730-47cc-95f4-1493a726d645-tls-certs\") pod \"4f680056-7730-47cc-95f4-1493a726d645\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " Nov 02 08:22:32.622382 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.622295 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-tmp-dir" pod "4f680056-7730-47cc-95f4-1493a726d645" (UID: "4f680056-7730-47cc-95f4-1493a726d645"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:32.622382 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.622309 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-home\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-home\") pod \"4f680056-7730-47cc-95f4-1493a726d645\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " Nov 02 08:22:32.622382 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.622345 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-dshm\") pod \"4f680056-7730-47cc-95f4-1493a726d645\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " Nov 02 08:22:32.622382 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.622374 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-model-cache" pod "4f680056-7730-47cc-95f4-1493a726d645" (UID: "4f680056-7730-47cc-95f4-1493a726d645"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:32.622593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.622450 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f680056-7730-47cc-95f4-1493a726d645-kube-api-access-sk2zk\" (UniqueName: \"kubernetes.io/projected/4f680056-7730-47cc-95f4-1493a726d645-kube-api-access-sk2zk\") pod \"4f680056-7730-47cc-95f4-1493a726d645\" (UID: \"4f680056-7730-47cc-95f4-1493a726d645\") " Nov 02 08:22:32.622593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.622467 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-home" pod "4f680056-7730-47cc-95f4-1493a726d645" (UID: "4f680056-7730-47cc-95f4-1493a726d645"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:32.623035 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.623014 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:32.623185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.623037 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:32.623185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.623051 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:32.624576 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.624554 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-dshm" pod "4f680056-7730-47cc-95f4-1493a726d645" (UID: "4f680056-7730-47cc-95f4-1493a726d645"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:32.624669 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.624593 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f680056-7730-47cc-95f4-1493a726d645-kube-api-access-sk2zk" pod "4f680056-7730-47cc-95f4-1493a726d645" (UID: "4f680056-7730-47cc-95f4-1493a726d645"). InnerVolumeSpecName "kube-api-access-sk2zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:22:32.624669 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.624613 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f680056-7730-47cc-95f4-1493a726d645-tls-certs" pod "4f680056-7730-47cc-95f4-1493a726d645" (UID: "4f680056-7730-47cc-95f4-1493a726d645"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:22:32.724403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.724327 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f680056-7730-47cc-95f4-1493a726d645-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:32.724403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.724356 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk2zk\" (UniqueName: \"kubernetes.io/projected/4f680056-7730-47cc-95f4-1493a726d645-kube-api-access-sk2zk\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:32.724403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.724367 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f680056-7730-47cc-95f4-1493a726d645-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:32.829637 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.829599 5899 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 08:22:32.829820 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:32.829653 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" podUID="57768ba3-8718-4280-8205-1827026315b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 08:22:33.019752 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.019724 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:22:33.047627 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.047590 5899 generic.go:356] "Generic (PLEG): container finished" podID="4f680056-7730-47cc-95f4-1493a726d645" containerID="55f1ca60a2adc06194b7546590969d0d46a5ff457ee2e660290e3e7f284b7cb8" exitCode=0 Nov 02 08:22:33.047824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.047633 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" Nov 02 08:22:33.047824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.047677 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" event={"ID":"4f680056-7730-47cc-95f4-1493a726d645","Type":"ContainerDied","Data":"55f1ca60a2adc06194b7546590969d0d46a5ff457ee2e660290e3e7f284b7cb8"} Nov 02 08:22:33.047824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.047716 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2" event={"ID":"4f680056-7730-47cc-95f4-1493a726d645","Type":"ContainerDied","Data":"1dde26a83fbde3748fa1bbd22978783075a77b7b20ce6a52977d150b15625ade"} Nov 02 08:22:33.047824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.047737 5899 scope.go:122] "RemoveContainer" containerID="55f1ca60a2adc06194b7546590969d0d46a5ff457ee2e660290e3e7f284b7cb8" Nov 02 08:22:33.049336 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.049308 5899 generic.go:356] "Generic (PLEG): container finished" podID="57768ba3-8718-4280-8205-1827026315b9" containerID="8c9694f65a210fc374a040ee254de633b63ba60cb8360e0d7df37b5a9d5e7b76" exitCode=0 Nov 02 08:22:33.049441 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.049339 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" Nov 02 08:22:33.049441 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.049374 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" event={"ID":"57768ba3-8718-4280-8205-1827026315b9","Type":"ContainerDied","Data":"8c9694f65a210fc374a040ee254de633b63ba60cb8360e0d7df37b5a9d5e7b76"} Nov 02 08:22:33.049441 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.049420 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq" event={"ID":"57768ba3-8718-4280-8205-1827026315b9","Type":"ContainerDied","Data":"9a3d70786b1f8c96fe49fb02a05960691285f807704d097ba4f8c4d9200f9096"} Nov 02 08:22:33.058385 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.058361 5899 scope.go:122] "RemoveContainer" containerID="55f1ca60a2adc06194b7546590969d0d46a5ff457ee2e660290e3e7f284b7cb8" Nov 02 08:22:33.058657 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:22:33.058637 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f1ca60a2adc06194b7546590969d0d46a5ff457ee2e660290e3e7f284b7cb8\": container with ID starting with 55f1ca60a2adc06194b7546590969d0d46a5ff457ee2e660290e3e7f284b7cb8 not found: ID does not exist" containerID="55f1ca60a2adc06194b7546590969d0d46a5ff457ee2e660290e3e7f284b7cb8" Nov 02 08:22:33.058762 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.058671 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f1ca60a2adc06194b7546590969d0d46a5ff457ee2e660290e3e7f284b7cb8"} err="failed to get container status \"55f1ca60a2adc06194b7546590969d0d46a5ff457ee2e660290e3e7f284b7cb8\": rpc error: code = NotFound desc = could not find container \"55f1ca60a2adc06194b7546590969d0d46a5ff457ee2e660290e3e7f284b7cb8\": container with ID starting with 55f1ca60a2adc06194b7546590969d0d46a5ff457ee2e660290e3e7f284b7cb8 not found: ID does not exist" Nov 02 08:22:33.058762 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.058698 5899 scope.go:122] "RemoveContainer" containerID="8c9694f65a210fc374a040ee254de633b63ba60cb8360e0d7df37b5a9d5e7b76" Nov 02 08:22:33.068922 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.068898 5899 scope.go:122] "RemoveContainer" containerID="8c9694f65a210fc374a040ee254de633b63ba60cb8360e0d7df37b5a9d5e7b76" Nov 02 08:22:33.069399 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:22:33.069366 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9694f65a210fc374a040ee254de633b63ba60cb8360e0d7df37b5a9d5e7b76\": container with ID starting with 8c9694f65a210fc374a040ee254de633b63ba60cb8360e0d7df37b5a9d5e7b76 not found: ID does not exist" containerID="8c9694f65a210fc374a040ee254de633b63ba60cb8360e0d7df37b5a9d5e7b76" Nov 02 08:22:33.069496 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.069407 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9694f65a210fc374a040ee254de633b63ba60cb8360e0d7df37b5a9d5e7b76"} err="failed to get container status \"8c9694f65a210fc374a040ee254de633b63ba60cb8360e0d7df37b5a9d5e7b76\": rpc error: code = NotFound desc = could not find container \"8c9694f65a210fc374a040ee254de633b63ba60cb8360e0d7df37b5a9d5e7b76\": container with ID starting with 8c9694f65a210fc374a040ee254de633b63ba60cb8360e0d7df37b5a9d5e7b76 not found: ID does not exist" Nov 02 08:22:33.106593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.106560 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2"] Nov 02 08:22:33.113886 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.113853 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f765b8/test-model-f765b8-kserve-6c97ccf9f9-qngv2"] Nov 02 08:22:33.130050 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.130025 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-istio-token\" (UniqueName: \"kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-istio-token\") pod \"57768ba3-8718-4280-8205-1827026315b9\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " Nov 02 08:22:33.130202 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.130089 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-kube-api-access-xcqgn\" (UniqueName: \"kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-kube-api-access-xcqgn\") pod \"57768ba3-8718-4280-8205-1827026315b9\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " Nov 02 08:22:33.130202 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.130134 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-certs\") pod \"57768ba3-8718-4280-8205-1827026315b9\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " Nov 02 08:22:33.130202 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.130154 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-data\") pod \"57768ba3-8718-4280-8205-1827026315b9\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " Nov 02 08:22:33.130381 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.130267 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57768ba3-8718-4280-8205-1827026315b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57768ba3-8718-4280-8205-1827026315b9-istio-podinfo\") pod \"57768ba3-8718-4280-8205-1827026315b9\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " Nov 02 08:22:33.130381 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.130310 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-envoy\") pod \"57768ba3-8718-4280-8205-1827026315b9\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " Nov 02 08:22:33.130488 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.130422 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-credential-socket\") pod \"57768ba3-8718-4280-8205-1827026315b9\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " Nov 02 08:22:33.130542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.130487 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-certs" pod "57768ba3-8718-4280-8205-1827026315b9" (UID: "57768ba3-8718-4280-8205-1827026315b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:33.130542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.130514 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-socket\") pod \"57768ba3-8718-4280-8205-1827026315b9\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " Nov 02 08:22:33.130660 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.130537 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-data" pod "57768ba3-8718-4280-8205-1827026315b9" (UID: "57768ba3-8718-4280-8205-1827026315b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:33.130660 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.130613 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57768ba3-8718-4280-8205-1827026315b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57768ba3-8718-4280-8205-1827026315b9-istiod-ca-cert\") pod \"57768ba3-8718-4280-8205-1827026315b9\" (UID: \"57768ba3-8718-4280-8205-1827026315b9\") " Nov 02 08:22:33.130863 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.130836 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-credential-socket" pod "57768ba3-8718-4280-8205-1827026315b9" (UID: "57768ba3-8718-4280-8205-1827026315b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:33.130963 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.130943 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-socket" pod "57768ba3-8718-4280-8205-1827026315b9" (UID: "57768ba3-8718-4280-8205-1827026315b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:33.131046 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.131019 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57768ba3-8718-4280-8205-1827026315b9-istiod-ca-cert" pod "57768ba3-8718-4280-8205-1827026315b9" (UID: "57768ba3-8718-4280-8205-1827026315b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:22:33.131264 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.131246 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:33.131344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.131264 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:33.131344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.131275 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57768ba3-8718-4280-8205-1827026315b9-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:33.131344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.131284 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:33.131344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.131292 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:33.132619 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.132595 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57768ba3-8718-4280-8205-1827026315b9-istio-podinfo" pod "57768ba3-8718-4280-8205-1827026315b9" (UID: "57768ba3-8718-4280-8205-1827026315b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:22:33.132717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.132600 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-istio-token" pod "57768ba3-8718-4280-8205-1827026315b9" (UID: "57768ba3-8718-4280-8205-1827026315b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:22:33.132769 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.132753 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-envoy" pod "57768ba3-8718-4280-8205-1827026315b9" (UID: "57768ba3-8718-4280-8205-1827026315b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:33.132769 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.132759 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-kube-api-access-xcqgn" pod "57768ba3-8718-4280-8205-1827026315b9" (UID: "57768ba3-8718-4280-8205-1827026315b9"). InnerVolumeSpecName "kube-api-access-xcqgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:22:33.233168 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.233124 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57768ba3-8718-4280-8205-1827026315b9-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:33.233168 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.233152 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57768ba3-8718-4280-8205-1827026315b9-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:33.233540 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.233189 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:33.233540 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.233198 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcqgn\" (UniqueName: \"kubernetes.io/projected/57768ba3-8718-4280-8205-1827026315b9-kube-api-access-xcqgn\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:33.399146 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.399108 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq"] Nov 02 08:22:33.407919 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.407889 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-fr8zq"] Nov 02 08:22:33.820605 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.820575 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f680056-7730-47cc-95f4-1493a726d645" path="/var/lib/kubelet/pods/4f680056-7730-47cc-95f4-1493a726d645/volumes" Nov 02 08:22:33.821083 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:33.821070 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57768ba3-8718-4280-8205-1827026315b9" path="/var/lib/kubelet/pods/57768ba3-8718-4280-8205-1827026315b9/volumes" Nov 02 08:22:34.017015 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:34.016975 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:35.881266 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:35.881231 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:37.739794 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:37.739759 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:39.203227 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.203132 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" podUID="97ce128f-f1ec-4ecf-904d-ed9481ff6e0b" containerName="main" containerID="cri-o://9d2ec49b01307d68c043524555155c548e56d14d380ed46f7a1e8e5a7f0c35c8" gracePeriod=45 Nov 02 08:22:39.512823 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.512783 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:39.515915 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.515891 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:22:39.614181 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.614117 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-dshm\") pod \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " Nov 02 08:22:39.614389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.614228 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tmp-dir\") pod \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " Nov 02 08:22:39.614389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.614366 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-home\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-home\") pod \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " Nov 02 08:22:39.614519 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.614468 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tls-certs\") pod \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " Nov 02 08:22:39.614519 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.614513 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-kube-api-access-s9757\" (UniqueName: \"kubernetes.io/projected/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-kube-api-access-s9757\") pod \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " Nov 02 08:22:39.614634 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.614543 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-model-cache\") pod \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\" (UID: \"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b\") " Nov 02 08:22:39.614634 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.614582 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tmp-dir" pod "97ce128f-f1ec-4ecf-904d-ed9481ff6e0b" (UID: "97ce128f-f1ec-4ecf-904d-ed9481ff6e0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:39.615131 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.614912 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-home" pod "97ce128f-f1ec-4ecf-904d-ed9481ff6e0b" (UID: "97ce128f-f1ec-4ecf-904d-ed9481ff6e0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:39.615131 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.615097 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-model-cache" pod "97ce128f-f1ec-4ecf-904d-ed9481ff6e0b" (UID: "97ce128f-f1ec-4ecf-904d-ed9481ff6e0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:39.615347 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.615305 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:39.615347 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.615329 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:39.615347 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.615343 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:39.616781 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.616755 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-dshm" pod "97ce128f-f1ec-4ecf-904d-ed9481ff6e0b" (UID: "97ce128f-f1ec-4ecf-904d-ed9481ff6e0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:22:39.617113 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.617091 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tls-certs" pod "97ce128f-f1ec-4ecf-904d-ed9481ff6e0b" (UID: "97ce128f-f1ec-4ecf-904d-ed9481ff6e0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:22:39.617220 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.617113 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-kube-api-access-s9757" pod "97ce128f-f1ec-4ecf-904d-ed9481ff6e0b" (UID: "97ce128f-f1ec-4ecf-904d-ed9481ff6e0b"). InnerVolumeSpecName "kube-api-access-s9757". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:22:39.716666 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.716633 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:39.716666 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.716663 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:39.716877 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:39.716672 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9757\" (UniqueName: \"kubernetes.io/projected/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b-kube-api-access-s9757\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:40.103229 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:40.103130 5899 generic.go:356] "Generic (PLEG): container finished" podID="97ce128f-f1ec-4ecf-904d-ed9481ff6e0b" containerID="9d2ec49b01307d68c043524555155c548e56d14d380ed46f7a1e8e5a7f0c35c8" exitCode=0 Nov 02 08:22:40.103229 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:40.103204 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" Nov 02 08:22:40.103456 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:40.103219 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" event={"ID":"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b","Type":"ContainerDied","Data":"9d2ec49b01307d68c043524555155c548e56d14d380ed46f7a1e8e5a7f0c35c8"} Nov 02 08:22:40.103456 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:40.103274 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr" event={"ID":"97ce128f-f1ec-4ecf-904d-ed9481ff6e0b","Type":"ContainerDied","Data":"4be723b4d51effdbba5f4a8570a04f3e0fb8a4d4f49575a0e447faa1bddff159"} Nov 02 08:22:40.103456 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:40.103295 5899 scope.go:122] "RemoveContainer" containerID="9d2ec49b01307d68c043524555155c548e56d14d380ed46f7a1e8e5a7f0c35c8" Nov 02 08:22:40.113124 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:40.113100 5899 scope.go:122] "RemoveContainer" containerID="9d2ec49b01307d68c043524555155c548e56d14d380ed46f7a1e8e5a7f0c35c8" Nov 02 08:22:40.113418 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:22:40.113396 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2ec49b01307d68c043524555155c548e56d14d380ed46f7a1e8e5a7f0c35c8\": container with ID starting with 9d2ec49b01307d68c043524555155c548e56d14d380ed46f7a1e8e5a7f0c35c8 not found: ID does not exist" containerID="9d2ec49b01307d68c043524555155c548e56d14d380ed46f7a1e8e5a7f0c35c8" Nov 02 08:22:40.113495 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:40.113431 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2ec49b01307d68c043524555155c548e56d14d380ed46f7a1e8e5a7f0c35c8"} err="failed to get container status \"9d2ec49b01307d68c043524555155c548e56d14d380ed46f7a1e8e5a7f0c35c8\": rpc error: code = NotFound desc = could not find container \"9d2ec49b01307d68c043524555155c548e56d14d380ed46f7a1e8e5a7f0c35c8\": container with ID starting with 9d2ec49b01307d68c043524555155c548e56d14d380ed46f7a1e8e5a7f0c35c8 not found: ID does not exist" Nov 02 08:22:40.158593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:40.158566 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr"] Nov 02 08:22:40.174292 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:40.174239 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c4489d/test-model-c4489d-kserve-5fb864c89f-dvzbr"] Nov 02 08:22:40.346502 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:40.346467 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:40.492408 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:40.492373 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:41.315066 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:41.315027 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:41.820253 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:41.820217 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ce128f-f1ec-4ecf-904d-ed9481ff6e0b" path="/var/lib/kubelet/pods/97ce128f-f1ec-4ecf-904d-ed9481ff6e0b/volumes" Nov 02 08:22:42.864819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:42.864783 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb"] Nov 02 08:22:42.865369 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:42.864990 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" podUID="3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5" containerName="maas-api" containerID="cri-o://3f774c96518934619fa47985f44fa47e177e9c4bb8e747abc94e1c235dfadbd3" gracePeriod=30 Nov 02 08:22:42.967831 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:42.967779 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:43.144400 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.137194 5899 generic.go:356] "Generic (PLEG): container finished" podID="3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5" containerID="3f774c96518934619fa47985f44fa47e177e9c4bb8e747abc94e1c235dfadbd3" exitCode=0 Nov 02 08:22:43.144400 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.137399 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" event={"ID":"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5","Type":"ContainerDied","Data":"3f774c96518934619fa47985f44fa47e177e9c4bb8e747abc94e1c235dfadbd3"} Nov 02 08:22:43.157520 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:22:43.154048 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:22:43.185379 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.185349 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:22:43.269257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.269214 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-maas-api-tls\") pod \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " Nov 02 08:22:43.269462 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.269349 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-openshift-service-ca\") pod \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " Nov 02 08:22:43.269462 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.269383 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-kube-api-access-zscjb\" (UniqueName: \"kubernetes.io/projected/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-kube-api-access-zscjb\") pod \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " Nov 02 08:22:43.269575 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.269561 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-metrics-certs\") pod \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\" (UID: \"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5\") " Nov 02 08:22:43.269950 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.269866 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-openshift-service-ca" pod "3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5" (UID: "3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:22:43.270524 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.270308 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:43.272108 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.272080 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-maas-api-tls" pod "3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5" (UID: "3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:22:43.272786 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.272761 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-metrics-certs" pod "3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5" (UID: "3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:22:43.273052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.273026 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-kube-api-access-zscjb" pod "3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5" (UID: "3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5"). InnerVolumeSpecName "kube-api-access-zscjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:22:43.371943 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.371909 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zscjb\" (UniqueName: \"kubernetes.io/projected/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-kube-api-access-zscjb\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:43.371943 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.371943 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:43.372188 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.371953 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:22:43.764642 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:43.764605 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:44.146048 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:44.145965 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" Nov 02 08:22:44.146048 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:44.146004 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb" event={"ID":"3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5","Type":"ContainerDied","Data":"ec69e52c34e50c923fe27500678d2d8e49f363094a0858db60364151ece04cc3"} Nov 02 08:22:44.146567 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:44.146079 5899 scope.go:122] "RemoveContainer" containerID="3f774c96518934619fa47985f44fa47e177e9c4bb8e747abc94e1c235dfadbd3" Nov 02 08:22:44.223790 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:44.223754 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb"] Nov 02 08:22:44.236797 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:44.236762 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e271b1-5dc9fcffb5-pgssb"] Nov 02 08:22:45.821418 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:45.821385 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5" path="/var/lib/kubelet/pods/3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5/volumes" Nov 02 08:22:45.977604 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:45.977570 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:47.286528 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:47.286485 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq"] Nov 02 08:22:47.431601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:47.431562 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:47.884778 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:47.884721 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:48.051417 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:48.051384 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:48.385850 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:48.385799 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:48.610903 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:48.610872 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:49.292365 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:49.292323 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:49.721029 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:49.720928 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:50.324547 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:50.324512 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:50.645657 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:50.645576 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:52.424698 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:52.424659 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:53.359488 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:53.359452 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:53.527480 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:53.527448 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:54.713713 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:54.713681 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:57.121056 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:57.121021 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:57.420343 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:57.420265 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:59.163757 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:59.163715 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:59.324203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:59.324154 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:22:59.708345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:22:59.708311 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:00.955870 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:00.955834 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669"] Nov 02 08:23:00.956286 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:00.956040 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" podUID="8e4269e0-a8c4-40fa-90c6-34a67ac7e810" containerName="maas-api" containerID="cri-o://a3abf0bf4c3798fe0b91650653f999e8920ca1c066a8dca9cbc28ceb7b372537" gracePeriod=30 Nov 02 08:23:01.072026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.071997 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:01.274520 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.274486 5899 generic.go:356] "Generic (PLEG): container finished" podID="8e4269e0-a8c4-40fa-90c6-34a67ac7e810" containerID="a3abf0bf4c3798fe0b91650653f999e8920ca1c066a8dca9cbc28ceb7b372537" exitCode=0 Nov 02 08:23:01.274766 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.274745 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" event={"ID":"8e4269e0-a8c4-40fa-90c6-34a67ac7e810","Type":"ContainerDied","Data":"a3abf0bf4c3798fe0b91650653f999e8920ca1c066a8dca9cbc28ceb7b372537"} Nov 02 08:23:01.274929 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.274909 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" event={"ID":"8e4269e0-a8c4-40fa-90c6-34a67ac7e810","Type":"ContainerDied","Data":"5ae60f64192b18dc937121c4f4fd0bfe8b8c954d19acfc4c0aa391eb54a03150"} Nov 02 08:23:01.275073 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.275050 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ae60f64192b18dc937121c4f4fd0bfe8b8c954d19acfc4c0aa391eb54a03150" Nov 02 08:23:01.277435 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.277414 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:23:01.320722 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.320631 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-metrics-certs\") pod \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " Nov 02 08:23:01.320889 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.320753 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-kube-api-access-mmxzl\" (UniqueName: \"kubernetes.io/projected/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-kube-api-access-mmxzl\") pod \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " Nov 02 08:23:01.320889 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.320883 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-openshift-service-ca\") pod \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " Nov 02 08:23:01.321021 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.320963 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-maas-api-tls\") pod \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\" (UID: \"8e4269e0-a8c4-40fa-90c6-34a67ac7e810\") " Nov 02 08:23:01.321310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.321285 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-openshift-service-ca" pod "8e4269e0-a8c4-40fa-90c6-34a67ac7e810" (UID: "8e4269e0-a8c4-40fa-90c6-34a67ac7e810"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:23:01.321585 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.321568 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:01.323217 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.323146 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-metrics-certs" pod "8e4269e0-a8c4-40fa-90c6-34a67ac7e810" (UID: "8e4269e0-a8c4-40fa-90c6-34a67ac7e810"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:23:01.323641 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.323618 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-kube-api-access-mmxzl" pod "8e4269e0-a8c4-40fa-90c6-34a67ac7e810" (UID: "8e4269e0-a8c4-40fa-90c6-34a67ac7e810"). InnerVolumeSpecName "kube-api-access-mmxzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:23:01.323726 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.323704 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-maas-api-tls" pod "8e4269e0-a8c4-40fa-90c6-34a67ac7e810" (UID: "8e4269e0-a8c4-40fa-90c6-34a67ac7e810"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:23:01.422649 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.422610 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:01.422649 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.422641 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmxzl\" (UniqueName: \"kubernetes.io/projected/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-kube-api-access-mmxzl\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:01.422649 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.422657 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e4269e0-a8c4-40fa-90c6-34a67ac7e810-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:01.988344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:01.988309 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:02.281062 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.281016 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669" Nov 02 08:23:02.308406 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.308371 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" podUID="4c2f5b4f-7f8e-426c-a9e7-92aca079659d" containerName="main" containerID="cri-o://22c36d9d9c6edddfbf02e22186246d3a7ff4343c086e85652c5b1f36b8131b4d" gracePeriod=45 Nov 02 08:23:02.339106 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.339076 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669"] Nov 02 08:23:02.352000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.351973 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f765b8-74f9b8f64-4d669"] Nov 02 08:23:02.595081 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.595051 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:23:02.638974 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.638936 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-model-cache\") pod \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " Nov 02 08:23:02.639215 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.639039 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tmp-dir\") pod \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " Nov 02 08:23:02.639215 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.639128 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-dshm\") pod \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " Nov 02 08:23:02.639215 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.639191 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-kube-api-access-rtrn2\" (UniqueName: \"kubernetes.io/projected/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-kube-api-access-rtrn2\") pod \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " Nov 02 08:23:02.639389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.639234 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tls-certs\") pod \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " Nov 02 08:23:02.639389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.639326 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-model-cache" pod "4c2f5b4f-7f8e-426c-a9e7-92aca079659d" (UID: "4c2f5b4f-7f8e-426c-a9e7-92aca079659d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:23:02.639389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.639344 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-home\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-home\") pod \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\" (UID: \"4c2f5b4f-7f8e-426c-a9e7-92aca079659d\") " Nov 02 08:23:02.639541 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.639342 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tmp-dir" pod "4c2f5b4f-7f8e-426c-a9e7-92aca079659d" (UID: "4c2f5b4f-7f8e-426c-a9e7-92aca079659d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:23:02.639598 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.639537 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-home" pod "4c2f5b4f-7f8e-426c-a9e7-92aca079659d" (UID: "4c2f5b4f-7f8e-426c-a9e7-92aca079659d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:23:02.639908 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.639884 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:02.640033 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.639933 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:02.640033 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.639945 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:02.643537 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.641986 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tls-certs" pod "4c2f5b4f-7f8e-426c-a9e7-92aca079659d" (UID: "4c2f5b4f-7f8e-426c-a9e7-92aca079659d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:23:02.643537 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.642012 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-dshm" pod "4c2f5b4f-7f8e-426c-a9e7-92aca079659d" (UID: "4c2f5b4f-7f8e-426c-a9e7-92aca079659d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:23:02.643537 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.642188 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-kube-api-access-rtrn2" pod "4c2f5b4f-7f8e-426c-a9e7-92aca079659d" (UID: "4c2f5b4f-7f8e-426c-a9e7-92aca079659d"). InnerVolumeSpecName "kube-api-access-rtrn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:23:02.741466 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.741430 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:02.741466 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.741461 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtrn2\" (UniqueName: \"kubernetes.io/projected/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-kube-api-access-rtrn2\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:02.741466 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:02.741474 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2f5b4f-7f8e-426c-a9e7-92aca079659d-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:03.289910 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:03.289870 5899 generic.go:356] "Generic (PLEG): container finished" podID="4c2f5b4f-7f8e-426c-a9e7-92aca079659d" containerID="22c36d9d9c6edddfbf02e22186246d3a7ff4343c086e85652c5b1f36b8131b4d" exitCode=0 Nov 02 08:23:03.289910 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:03.289912 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" Nov 02 08:23:03.290500 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:03.289948 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" event={"ID":"4c2f5b4f-7f8e-426c-a9e7-92aca079659d","Type":"ContainerDied","Data":"22c36d9d9c6edddfbf02e22186246d3a7ff4343c086e85652c5b1f36b8131b4d"} Nov 02 08:23:03.290500 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:03.290001 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq" event={"ID":"4c2f5b4f-7f8e-426c-a9e7-92aca079659d","Type":"ContainerDied","Data":"6ed862d17d19ccae293c1e5b349d84961add92732eee7623d7f2a48c1bd9bf5e"} Nov 02 08:23:03.290500 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:03.290023 5899 scope.go:122] "RemoveContainer" containerID="22c36d9d9c6edddfbf02e22186246d3a7ff4343c086e85652c5b1f36b8131b4d" Nov 02 08:23:03.301154 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:03.301134 5899 scope.go:122] "RemoveContainer" containerID="22c36d9d9c6edddfbf02e22186246d3a7ff4343c086e85652c5b1f36b8131b4d" Nov 02 08:23:03.301473 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:23:03.301454 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c36d9d9c6edddfbf02e22186246d3a7ff4343c086e85652c5b1f36b8131b4d\": container with ID starting with 22c36d9d9c6edddfbf02e22186246d3a7ff4343c086e85652c5b1f36b8131b4d not found: ID does not exist" containerID="22c36d9d9c6edddfbf02e22186246d3a7ff4343c086e85652c5b1f36b8131b4d" Nov 02 08:23:03.301532 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:03.301483 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c36d9d9c6edddfbf02e22186246d3a7ff4343c086e85652c5b1f36b8131b4d"} err="failed to get container status \"22c36d9d9c6edddfbf02e22186246d3a7ff4343c086e85652c5b1f36b8131b4d\": rpc error: code = NotFound desc = could not find container \"22c36d9d9c6edddfbf02e22186246d3a7ff4343c086e85652c5b1f36b8131b4d\": container with ID starting with 22c36d9d9c6edddfbf02e22186246d3a7ff4343c086e85652c5b1f36b8131b4d not found: ID does not exist" Nov 02 08:23:03.345001 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:03.344975 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq"] Nov 02 08:23:03.361204 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:03.361155 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-dfd6f0/test-model-dfd6f0-kserve-759f6d9c5-xppxq"] Nov 02 08:23:03.733375 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:03.733336 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:03.820743 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:03.820705 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2f5b4f-7f8e-426c-a9e7-92aca079659d" path="/var/lib/kubelet/pods/4c2f5b4f-7f8e-426c-a9e7-92aca079659d/volumes" Nov 02 08:23:03.821268 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:03.821253 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4269e0-a8c4-40fa-90c6-34a67ac7e810" path="/var/lib/kubelet/pods/8e4269e0-a8c4-40fa-90c6-34a67ac7e810/volumes" Nov 02 08:23:03.964967 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:03.964930 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:06.015130 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:06.015084 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:06.956387 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:06.956350 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:07.205818 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:07.205787 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:07.741460 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:07.741414 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:08.087484 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:08.087405 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:08.286486 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:08.286448 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:09.235994 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:09.235959 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:09.393887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:09.393851 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:10.687927 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:10.687888 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:12.600654 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:12.600602 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:13.142024 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:23:13.141986 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:23:14.475738 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:14.475702 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:16.421420 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:16.421388 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:17.036744 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:17.036691 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:17.287475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:17.287355 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:18.245045 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:18.245012 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:20.191838 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:20.191801 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:21.884351 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:21.884320 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:23.966896 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:23.966846 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:25.806947 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:25.806907 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:25.962717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:25.962682 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh"] Nov 02 08:23:26.451170 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:26.451113 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" podUID="9386adea-631b-4d70-a3f6-243b4ce8836d" containerName="istio-proxy" containerID="cri-o://a3e273a8f0aca0b4007f3d201116c9c799cc18779e454cccaef35b00fc36ad4d" gracePeriod=30 Nov 02 08:23:27.739137 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:27.739099 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:29.535789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:29.535758 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:31.332904 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.332873 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:31.728120 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.728092 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:23:31.816721 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.816690 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-credential-socket\") pod \"9386adea-631b-4d70-a3f6-243b4ce8836d\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " Nov 02 08:23:31.816894 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.816754 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-certs\") pod \"9386adea-631b-4d70-a3f6-243b4ce8836d\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " Nov 02 08:23:31.816894 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.816778 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-socket\") pod \"9386adea-631b-4d70-a3f6-243b4ce8836d\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " Nov 02 08:23:31.816894 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.816801 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-envoy\") pod \"9386adea-631b-4d70-a3f6-243b4ce8836d\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " Nov 02 08:23:31.816894 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.816872 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-data\") pod \"9386adea-631b-4d70-a3f6-243b4ce8836d\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " Nov 02 08:23:31.816894 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.816894 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9386adea-631b-4d70-a3f6-243b4ce8836d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9386adea-631b-4d70-a3f6-243b4ce8836d-istiod-ca-cert\") pod \"9386adea-631b-4d70-a3f6-243b4ce8836d\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " Nov 02 08:23:31.817082 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.816951 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-token\" (UniqueName: \"kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-token\") pod \"9386adea-631b-4d70-a3f6-243b4ce8836d\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " Nov 02 08:23:31.817082 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.816993 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-kube-api-access-4r8mr\" (UniqueName: \"kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-kube-api-access-4r8mr\") pod \"9386adea-631b-4d70-a3f6-243b4ce8836d\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " Nov 02 08:23:31.817082 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.817049 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-podinfo\") pod \"9386adea-631b-4d70-a3f6-243b4ce8836d\" (UID: \"9386adea-631b-4d70-a3f6-243b4ce8836d\") " Nov 02 08:23:31.818611 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.818578 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-certs" pod "9386adea-631b-4d70-a3f6-243b4ce8836d" (UID: "9386adea-631b-4d70-a3f6-243b4ce8836d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:23:31.818762 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.818642 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-credential-socket" pod "9386adea-631b-4d70-a3f6-243b4ce8836d" (UID: "9386adea-631b-4d70-a3f6-243b4ce8836d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:23:31.818835 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.818788 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-socket" pod "9386adea-631b-4d70-a3f6-243b4ce8836d" (UID: "9386adea-631b-4d70-a3f6-243b4ce8836d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:23:31.819065 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.819007 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9386adea-631b-4d70-a3f6-243b4ce8836d-istiod-ca-cert" pod "9386adea-631b-4d70-a3f6-243b4ce8836d" (UID: "9386adea-631b-4d70-a3f6-243b4ce8836d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:23:31.819308 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.819278 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-data" pod "9386adea-631b-4d70-a3f6-243b4ce8836d" (UID: "9386adea-631b-4d70-a3f6-243b4ce8836d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:23:31.821275 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.821250 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-podinfo" pod "9386adea-631b-4d70-a3f6-243b4ce8836d" (UID: "9386adea-631b-4d70-a3f6-243b4ce8836d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:23:31.821410 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.821387 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-token" pod "9386adea-631b-4d70-a3f6-243b4ce8836d" (UID: "9386adea-631b-4d70-a3f6-243b4ce8836d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:23:31.821476 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.821459 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-envoy" pod "9386adea-631b-4d70-a3f6-243b4ce8836d" (UID: "9386adea-631b-4d70-a3f6-243b4ce8836d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:23:31.821697 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.821672 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-kube-api-access-4r8mr" pod "9386adea-631b-4d70-a3f6-243b4ce8836d" (UID: "9386adea-631b-4d70-a3f6-243b4ce8836d"). InnerVolumeSpecName "kube-api-access-4r8mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:23:31.919276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.919185 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:31.919276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.919217 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:31.919276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.919230 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:31.919276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.919242 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:31.919276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.919253 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:31.919276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.919264 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:31.919276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.919276 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9386adea-631b-4d70-a3f6-243b4ce8836d-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:31.919632 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.919287 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:31.919632 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:31.919299 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r8mr\" (UniqueName: \"kubernetes.io/projected/9386adea-631b-4d70-a3f6-243b4ce8836d-kube-api-access-4r8mr\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:23:32.495046 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:32.495014 5899 generic.go:356] "Generic (PLEG): container finished" podID="9386adea-631b-4d70-a3f6-243b4ce8836d" containerID="a3e273a8f0aca0b4007f3d201116c9c799cc18779e454cccaef35b00fc36ad4d" exitCode=0 Nov 02 08:23:32.495503 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:32.495063 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" Nov 02 08:23:32.495503 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:32.495064 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" event={"ID":"9386adea-631b-4d70-a3f6-243b4ce8836d","Type":"ContainerDied","Data":"a3e273a8f0aca0b4007f3d201116c9c799cc18779e454cccaef35b00fc36ad4d"} Nov 02 08:23:32.495503 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:32.495200 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh" event={"ID":"9386adea-631b-4d70-a3f6-243b4ce8836d","Type":"ContainerDied","Data":"2ab8c9560d71701b90d8f3b28a666cd6e7727bc871b520c042e37789bba00223"} Nov 02 08:23:32.495503 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:32.495233 5899 scope.go:122] "RemoveContainer" containerID="a3e273a8f0aca0b4007f3d201116c9c799cc18779e454cccaef35b00fc36ad4d" Nov 02 08:23:32.505516 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:32.505485 5899 scope.go:122] "RemoveContainer" containerID="a3e273a8f0aca0b4007f3d201116c9c799cc18779e454cccaef35b00fc36ad4d" Nov 02 08:23:32.505789 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:23:32.505771 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e273a8f0aca0b4007f3d201116c9c799cc18779e454cccaef35b00fc36ad4d\": container with ID starting with a3e273a8f0aca0b4007f3d201116c9c799cc18779e454cccaef35b00fc36ad4d not found: ID does not exist" containerID="a3e273a8f0aca0b4007f3d201116c9c799cc18779e454cccaef35b00fc36ad4d" Nov 02 08:23:32.505845 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:32.505799 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e273a8f0aca0b4007f3d201116c9c799cc18779e454cccaef35b00fc36ad4d"} err="failed to get container status \"a3e273a8f0aca0b4007f3d201116c9c799cc18779e454cccaef35b00fc36ad4d\": rpc error: code = NotFound desc = could not find container \"a3e273a8f0aca0b4007f3d201116c9c799cc18779e454cccaef35b00fc36ad4d\": container with ID starting with a3e273a8f0aca0b4007f3d201116c9c799cc18779e454cccaef35b00fc36ad4d not found: ID does not exist" Nov 02 08:23:32.563458 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:32.563422 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh"] Nov 02 08:23:32.576257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:32.576221 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxxrh"] Nov 02 08:23:33.116888 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:33.116821 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:33.820441 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:33.820400 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9386adea-631b-4d70-a3f6-243b4ce8836d" path="/var/lib/kubelet/pods/9386adea-631b-4d70-a3f6-243b4ce8836d/volumes" Nov 02 08:23:34.913687 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:34.913653 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:36.514541 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:36.514496 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:37.321566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:37.321496 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:39.013880 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:39.013847 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:39.405678 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:39.405578 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:39.553547 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:39.553486 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:40.201731 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:40.201697 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:40.636554 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:40.636507 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:41.240125 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:41.240088 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:41.410112 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:41.410073 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:43.010967 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:43.010929 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:43.141733 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:23:43.141701 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:23:44.733351 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:44.733250 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:46.436998 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:46.436962 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:47.041838 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:47.041804 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:47.186291 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:47.186256 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:48.036067 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:48.036033 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:48.444519 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:48.444423 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:50.201435 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:50.201380 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:51.926525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:51.926488 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:53.615125 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:53.615080 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:53.920253 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:53.920169 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:55.407252 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:55.407213 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:57.112455 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:57.112367 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:23:58.913861 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:23:58.913826 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:00.792898 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:00.792857 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:00.840620 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:00.840590 5899 scope.go:122] "RemoveContainer" containerID="a3abf0bf4c3798fe0b91650653f999e8920ca1c066a8dca9cbc28ceb7b372537" Nov 02 08:24:02.454459 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:02.454416 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:04.220128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:04.220095 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:06.006228 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:06.006192 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:07.695906 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:07.695866 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:08.194983 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:08.194947 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:09.685102 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:09.685065 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:11.531391 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:11.531348 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:13.141692 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:24:13.141651 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:24:13.337670 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:13.337629 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:15.136694 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:15.136653 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:17.024034 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:17.023965 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:17.372746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:17.372664 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:18.798892 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:18.798859 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:20.495442 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:20.495406 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:22.303639 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:22.303561 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:24.162568 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:24.162535 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:25.895715 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:25.895681 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:27.720993 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:27.720958 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:29.507961 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:29.507924 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:31.206929 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:31.206895 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:33.119228 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:33.119150 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:34.805614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:34.805578 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:36.581243 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:36.581193 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:38.429574 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:38.429541 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:39.847289 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:39.847238 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz"] Nov 02 08:24:39.847808 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:39.847477 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" podUID="181e9908-5351-46a9-9939-7559114bddd0" containerName="maas-api" containerID="cri-o://0c2d449d5c76e97935a8662f3622a873539e493d83c2bedcdef421168a1b22b9" gracePeriod=30 Nov 02 08:24:39.937898 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:39.937862 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:40.143370 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.143340 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:24:40.190693 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.189774 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181e9908-5351-46a9-9939-7559114bddd0-kube-api-access-ctc8w\" (UniqueName: \"kubernetes.io/projected/181e9908-5351-46a9-9939-7559114bddd0-kube-api-access-ctc8w\") pod \"181e9908-5351-46a9-9939-7559114bddd0\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " Nov 02 08:24:40.190693 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.189862 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-maas-api-tls\") pod \"181e9908-5351-46a9-9939-7559114bddd0\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " Nov 02 08:24:40.190693 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.189914 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-metrics-certs\") pod \"181e9908-5351-46a9-9939-7559114bddd0\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " Nov 02 08:24:40.190693 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.190092 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181e9908-5351-46a9-9939-7559114bddd0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/181e9908-5351-46a9-9939-7559114bddd0-openshift-service-ca\") pod \"181e9908-5351-46a9-9939-7559114bddd0\" (UID: \"181e9908-5351-46a9-9939-7559114bddd0\") " Nov 02 08:24:40.191080 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.190958 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181e9908-5351-46a9-9939-7559114bddd0-openshift-service-ca" pod "181e9908-5351-46a9-9939-7559114bddd0" (UID: "181e9908-5351-46a9-9939-7559114bddd0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:24:40.194572 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.194531 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-metrics-certs" pod "181e9908-5351-46a9-9939-7559114bddd0" (UID: "181e9908-5351-46a9-9939-7559114bddd0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:24:40.194572 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.194554 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-maas-api-tls" pod "181e9908-5351-46a9-9939-7559114bddd0" (UID: "181e9908-5351-46a9-9939-7559114bddd0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:24:40.194784 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.194579 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181e9908-5351-46a9-9939-7559114bddd0-kube-api-access-ctc8w" pod "181e9908-5351-46a9-9939-7559114bddd0" (UID: "181e9908-5351-46a9-9939-7559114bddd0"). InnerVolumeSpecName "kube-api-access-ctc8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:24:40.292636 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.292596 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/181e9908-5351-46a9-9939-7559114bddd0-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:24:40.292636 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.292633 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctc8w\" (UniqueName: \"kubernetes.io/projected/181e9908-5351-46a9-9939-7559114bddd0-kube-api-access-ctc8w\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:24:40.292636 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.292643 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:24:40.292898 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.292652 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/181e9908-5351-46a9-9939-7559114bddd0-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:24:40.648969 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.648933 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:40.957991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.957873 5899 generic.go:356] "Generic (PLEG): container finished" podID="181e9908-5351-46a9-9939-7559114bddd0" containerID="0c2d449d5c76e97935a8662f3622a873539e493d83c2bedcdef421168a1b22b9" exitCode=0 Nov 02 08:24:40.957991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.957951 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" event={"ID":"181e9908-5351-46a9-9939-7559114bddd0","Type":"ContainerDied","Data":"0c2d449d5c76e97935a8662f3622a873539e493d83c2bedcdef421168a1b22b9"} Nov 02 08:24:40.957991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.957918 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" Nov 02 08:24:40.958538 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.958012 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz" event={"ID":"181e9908-5351-46a9-9939-7559114bddd0","Type":"ContainerDied","Data":"81fe5725c1004f9f67a7b8237f55aa02ff909e41de695d5b446bd674dcd84ab0"} Nov 02 08:24:40.958538 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.958042 5899 scope.go:122] "RemoveContainer" containerID="0c2d449d5c76e97935a8662f3622a873539e493d83c2bedcdef421168a1b22b9" Nov 02 08:24:40.969452 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.969431 5899 scope.go:122] "RemoveContainer" containerID="0c2d449d5c76e97935a8662f3622a873539e493d83c2bedcdef421168a1b22b9" Nov 02 08:24:40.969750 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:24:40.969730 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2d449d5c76e97935a8662f3622a873539e493d83c2bedcdef421168a1b22b9\": container with ID starting with 0c2d449d5c76e97935a8662f3622a873539e493d83c2bedcdef421168a1b22b9 not found: ID does not exist" containerID="0c2d449d5c76e97935a8662f3622a873539e493d83c2bedcdef421168a1b22b9" Nov 02 08:24:40.969799 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:40.969758 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2d449d5c76e97935a8662f3622a873539e493d83c2bedcdef421168a1b22b9"} err="failed to get container status \"0c2d449d5c76e97935a8662f3622a873539e493d83c2bedcdef421168a1b22b9\": rpc error: code = NotFound desc = could not find container \"0c2d449d5c76e97935a8662f3622a873539e493d83c2bedcdef421168a1b22b9\": container with ID starting with 0c2d449d5c76e97935a8662f3622a873539e493d83c2bedcdef421168a1b22b9 not found: ID does not exist" Nov 02 08:24:41.007397 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.007357 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz"] Nov 02 08:24:41.019049 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.019017 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dfd6f0-7cf8986c78-zgrdz"] Nov 02 08:24:41.218772 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.218690 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm"] Nov 02 08:24:41.219661 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219637 5899 cpu_manager.go:414] "containerMap: removing container" podUID="3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5" containerName="maas-api" Nov 02 08:24:41.219776 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219663 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5" containerName="maas-api" Nov 02 08:24:41.219776 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219683 5899 cpu_manager.go:414] "containerMap: removing container" podUID="4c2f5b4f-7f8e-426c-a9e7-92aca079659d" containerName="main" Nov 02 08:24:41.219776 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219693 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2f5b4f-7f8e-426c-a9e7-92aca079659d" containerName="main" Nov 02 08:24:41.219776 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219710 5899 cpu_manager.go:414] "containerMap: removing container" podUID="9386adea-631b-4d70-a3f6-243b4ce8836d" containerName="istio-proxy" Nov 02 08:24:41.219776 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219717 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9386adea-631b-4d70-a3f6-243b4ce8836d" containerName="istio-proxy" Nov 02 08:24:41.219776 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219729 5899 cpu_manager.go:414] "containerMap: removing container" podUID="97ce128f-f1ec-4ecf-904d-ed9481ff6e0b" containerName="main" Nov 02 08:24:41.219776 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219736 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ce128f-f1ec-4ecf-904d-ed9481ff6e0b" containerName="main" Nov 02 08:24:41.219776 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219757 5899 cpu_manager.go:414] "containerMap: removing container" podUID="20d7be3b-a9c6-4a18-ae80-102530c29fdf" containerName="main" Nov 02 08:24:41.219776 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219765 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d7be3b-a9c6-4a18-ae80-102530c29fdf" containerName="main" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219794 5899 cpu_manager.go:414] "containerMap: removing container" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerName="istio-proxy" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219801 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerName="istio-proxy" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219812 5899 cpu_manager.go:414] "containerMap: removing container" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219819 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219846 5899 cpu_manager.go:414] "containerMap: removing container" podUID="8e4269e0-a8c4-40fa-90c6-34a67ac7e810" containerName="maas-api" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219853 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4269e0-a8c4-40fa-90c6-34a67ac7e810" containerName="maas-api" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219881 5899 cpu_manager.go:414] "containerMap: removing container" podUID="57768ba3-8718-4280-8205-1827026315b9" containerName="istio-proxy" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219887 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57768ba3-8718-4280-8205-1827026315b9" containerName="istio-proxy" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219897 5899 cpu_manager.go:414] "containerMap: removing container" podUID="4f680056-7730-47cc-95f4-1493a726d645" containerName="main" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219905 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f680056-7730-47cc-95f4-1493a726d645" containerName="main" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219920 5899 cpu_manager.go:414] "containerMap: removing container" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerName="istio-proxy" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219926 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerName="istio-proxy" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219935 5899 cpu_manager.go:414] "containerMap: removing container" podUID="181e9908-5351-46a9-9939-7559114bddd0" containerName="maas-api" Nov 02 08:24:41.220085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.219942 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181e9908-5351-46a9-9939-7559114bddd0" containerName="maas-api" Nov 02 08:24:41.220525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.220095 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6cad1f-c205-4e36-8c0a-3d9dd8d97bf5" containerName="maas-api" Nov 02 08:24:41.220525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.220109 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2f5b4f-7f8e-426c-a9e7-92aca079659d" containerName="main" Nov 02 08:24:41.220525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.220119 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="9386adea-631b-4d70-a3f6-243b4ce8836d" containerName="istio-proxy" Nov 02 08:24:41.220525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.220129 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ce128f-f1ec-4ecf-904d-ed9481ff6e0b" containerName="main" Nov 02 08:24:41.220525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.220141 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d7be3b-a9c6-4a18-ae80-102530c29fdf" containerName="main" Nov 02 08:24:41.220525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.220153 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cde1bc-a874-4ec0-b40c-c294de1a54bf" containerName="istio-proxy" Nov 02 08:24:41.220525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.220178 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ca22de-6df7-4d64-a7d3-4aa4653ce6b8" containerName="istio-proxy" Nov 02 08:24:41.220525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.220189 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4269e0-a8c4-40fa-90c6-34a67ac7e810" containerName="maas-api" Nov 02 08:24:41.220525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.220202 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="57768ba3-8718-4280-8205-1827026315b9" containerName="istio-proxy" Nov 02 08:24:41.220525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.220209 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f680056-7730-47cc-95f4-1493a726d645" containerName="main" Nov 02 08:24:41.220525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.220220 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0cefcf5-4b43-4286-8e0f-7bf2b228eeb9" containerName="istio-proxy" Nov 02 08:24:41.220525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.220229 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="181e9908-5351-46a9-9939-7559114bddd0" containerName="maas-api" Nov 02 08:24:41.225691 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.225670 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.280204 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.280142 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm"] Nov 02 08:24:41.304155 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.304120 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8af1193-18d1-42b6-b887-e96f00615754-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.304372 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.304181 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.304372 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.304278 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.304372 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.304312 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.304572 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.304398 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf7cl\" (UniqueName: \"kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-kube-api-access-qf7cl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.304572 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.304493 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8af1193-18d1-42b6-b887-e96f00615754-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.304572 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.304540 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.304668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.304608 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.304668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.304634 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.406341 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.406306 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8af1193-18d1-42b6-b887-e96f00615754-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.406511 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.406369 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.406511 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.406456 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.406511 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.406494 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.406624 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.406579 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8af1193-18d1-42b6-b887-e96f00615754-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.406624 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.406607 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.406703 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.406636 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.406703 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.406679 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.406797 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.406747 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf7cl\" (UniqueName: \"kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-kube-api-access-qf7cl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.406887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.406855 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.407063 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.407041 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.407199 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.407154 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.407288 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.407196 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8af1193-18d1-42b6-b887-e96f00615754-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.407335 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.407307 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.408774 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.408753 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.409076 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.409054 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8af1193-18d1-42b6-b887-e96f00615754-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.434861 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.434832 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.434973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.434944 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf7cl\" (UniqueName: \"kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-kube-api-access-qf7cl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2tbm\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.542657 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.542623 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:41.820498 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.820408 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181e9908-5351-46a9-9939-7559114bddd0" path="/var/lib/kubelet/pods/181e9908-5351-46a9-9939-7559114bddd0/volumes" Nov 02 08:24:41.863578 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.863552 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm"] Nov 02 08:24:41.865876 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:24:41.865848 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8af1193_18d1_42b6_b887_e96f00615754.slice/crio-4e865e80cc4e615633b4e001fddae244b871081028cd76d474f246396e3893e3 WatchSource:0}: Error finding container 4e865e80cc4e615633b4e001fddae244b871081028cd76d474f246396e3893e3: Status 404 returned error can't find the container with id 4e865e80cc4e615633b4e001fddae244b871081028cd76d474f246396e3893e3 Nov 02 08:24:41.868610 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.868578 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:24:41.868710 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.868643 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:24:41.868710 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.868672 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:24:41.967154 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.967119 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" event={"ID":"d8af1193-18d1-42b6-b887-e96f00615754","Type":"ContainerStarted","Data":"b7147a38e93cf9768ec3d9a3921852878a215e1480e8d36ec7a7c971dd065033"} Nov 02 08:24:41.967540 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:41.967243 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" event={"ID":"d8af1193-18d1-42b6-b887-e96f00615754","Type":"ContainerStarted","Data":"4e865e80cc4e615633b4e001fddae244b871081028cd76d474f246396e3893e3"} Nov 02 08:24:42.543425 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:42.543387 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:42.544687 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:42.544663 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 02 08:24:42.544799 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:42.544707 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" podUID="d8af1193-18d1-42b6-b887-e96f00615754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 02 08:24:42.595459 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:42.595412 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:43.141964 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:24:43.141928 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:24:43.543473 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:43.543430 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 02 08:24:43.543664 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:43.543489 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" podUID="d8af1193-18d1-42b6-b887-e96f00615754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 02 08:24:44.543128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:44.543091 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 02 08:24:44.543579 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:44.543143 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" podUID="d8af1193-18d1-42b6-b887-e96f00615754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 02 08:24:44.689852 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:44.689819 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:45.543356 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:45.543322 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 02 08:24:45.543759 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:45.543388 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" podUID="d8af1193-18d1-42b6-b887-e96f00615754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 02 08:24:46.139789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:46.139756 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:46.281632 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:46.281598 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:46.543905 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:46.543871 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 02 08:24:46.544328 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:46.543930 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" podUID="d8af1193-18d1-42b6-b887-e96f00615754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 02 08:24:46.828566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:46.828475 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:47.081418 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:47.081315 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:47.484688 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:47.477914 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:47.543834 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:47.543799 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 02 08:24:47.544081 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:47.544053 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" podUID="d8af1193-18d1-42b6-b887-e96f00615754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 02 08:24:48.320804 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:48.320769 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:48.458148 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:48.458117 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:48.543278 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:48.543239 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 02 08:24:48.543456 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:48.543309 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" podUID="d8af1193-18d1-42b6-b887-e96f00615754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 02 08:24:48.606049 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:48.605958 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:49.065192 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:49.065134 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:49.216872 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:49.216839 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:50.343944 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:50.343913 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:50.344462 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:50.344443 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:50.346627 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:50.346586 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:24:50.413472 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:50.413413 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" podStartSLOduration=9.413399274 podStartE2EDuration="9.413399274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:24:42.015582292 +0000 UTC m=+3582.313863120" watchObservedRunningTime="2026-11-02 08:24:50.413399274 +0000 UTC m=+3590.711680100" Nov 02 08:24:51.042837 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:51.042804 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:52.783146 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:52.783109 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:53.817753 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:53.817721 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:54.529876 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:54.529823 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:56.260196 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.260146 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm"] Nov 02 08:24:56.264762 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.264739 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.300261 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.300223 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm"] Nov 02 08:24:56.330072 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.330033 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c571f9-9dc1-493e-b1e5-a256cad13493-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.330279 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.330233 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7wjl\" (UniqueName: \"kubernetes.io/projected/b0c571f9-9dc1-493e-b1e5-a256cad13493-kube-api-access-m7wjl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.330391 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.330372 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c571f9-9dc1-493e-b1e5-a256cad13493-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.330452 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.330429 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.330510 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.330467 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c571f9-9dc1-493e-b1e5-a256cad13493-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.330583 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.330555 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.330644 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.330616 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.330792 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.330766 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.330870 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.330850 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.344405 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.344375 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:56.431887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.431847 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.432066 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.431898 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.432066 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.431962 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c571f9-9dc1-493e-b1e5-a256cad13493-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.432066 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.432011 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7wjl\" (UniqueName: \"kubernetes.io/projected/b0c571f9-9dc1-493e-b1e5-a256cad13493-kube-api-access-m7wjl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.432237 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.432072 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c571f9-9dc1-493e-b1e5-a256cad13493-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.432237 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.432093 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.432237 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.432126 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c571f9-9dc1-493e-b1e5-a256cad13493-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.432237 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.432175 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.432237 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.432211 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.432524 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.432367 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.432524 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.432400 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.432636 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.432532 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.432791 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.432762 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c571f9-9dc1-493e-b1e5-a256cad13493-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.432925 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.432802 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.434689 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.434660 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c571f9-9dc1-493e-b1e5-a256cad13493-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.434800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.434717 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c571f9-9dc1-493e-b1e5-a256cad13493-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.455098 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.455073 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c571f9-9dc1-493e-b1e5-a256cad13493-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.455367 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.455345 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7wjl\" (UniqueName: \"kubernetes.io/projected/b0c571f9-9dc1-493e-b1e5-a256cad13493-kube-api-access-m7wjl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cf7zm\" (UID: \"b0c571f9-9dc1-493e-b1e5-a256cad13493\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.580428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.580352 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:56.906029 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.906002 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm"] Nov 02 08:24:56.907097 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:24:56.907075 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c571f9_9dc1_493e_b1e5_a256cad13493.slice/crio-6a8568c4ec6b8c5aae8808e3d05defae02cd8d73be264211367609caf8cb9fcf WatchSource:0}: Error finding container 6a8568c4ec6b8c5aae8808e3d05defae02cd8d73be264211367609caf8cb9fcf: Status 404 returned error can't find the container with id 6a8568c4ec6b8c5aae8808e3d05defae02cd8d73be264211367609caf8cb9fcf Nov 02 08:24:56.910109 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.910070 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:24:56.910220 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.910157 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:24:56.910293 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:56.910215 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:24:57.080436 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:57.080398 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" event={"ID":"b0c571f9-9dc1-493e-b1e5-a256cad13493","Type":"ContainerStarted","Data":"562a78c8520b71df90c0de879a0f18c83570d578e9dbc08a408e986f751685e9"} Nov 02 08:24:57.080603 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:57.080459 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" event={"ID":"b0c571f9-9dc1-493e-b1e5-a256cad13493","Type":"ContainerStarted","Data":"6a8568c4ec6b8c5aae8808e3d05defae02cd8d73be264211367609caf8cb9fcf"} Nov 02 08:24:57.581509 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:57.581471 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:24:57.582944 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:57.582921 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cf7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:24:57.583004 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:57.582970 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" podUID="b0c571f9-9dc1-493e-b1e5-a256cad13493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 08:24:58.190149 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:58.190110 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:24:58.580822 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:58.580784 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cf7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:24:58.581039 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:58.580860 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" podUID="b0c571f9-9dc1-493e-b1e5-a256cad13493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 08:24:59.581772 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:59.581726 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cf7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:24:59.582428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:59.582380 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" podUID="b0c571f9-9dc1-493e-b1e5-a256cad13493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 08:24:59.942798 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:24:59.942687 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:00.084928 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:00.084891 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:00.581598 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:00.581560 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cf7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:25:00.581789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:00.581613 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" podUID="b0c571f9-9dc1-493e-b1e5-a256cad13493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 08:25:01.581621 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:01.581588 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cf7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:25:01.582006 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:01.581640 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" podUID="b0c571f9-9dc1-493e-b1e5-a256cad13493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 08:25:02.148428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:02.148393 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:02.580753 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:02.580712 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cf7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:25:02.581089 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:02.581039 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" podUID="b0c571f9-9dc1-493e-b1e5-a256cad13493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 08:25:03.581395 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:03.581355 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cf7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:25:03.581796 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:03.581433 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" podUID="b0c571f9-9dc1-493e-b1e5-a256cad13493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 08:25:03.981915 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:03.981879 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:04.581221 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:04.581153 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cf7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 08:25:04.581396 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:04.581247 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" podUID="b0c571f9-9dc1-493e-b1e5-a256cad13493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 08:25:05.704694 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:05.704662 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:25:05.705204 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:05.705179 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:25:05.706087 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:05.706065 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" Nov 02 08:25:05.762198 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:05.762128 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" podStartSLOduration=9.762115141 podStartE2EDuration="9.762115141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:24:57.124955376 +0000 UTC m=+3597.423236201" watchObservedRunningTime="2026-11-02 08:25:05.762115141 +0000 UTC m=+3606.060395966" Nov 02 08:25:05.885846 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:05.885813 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:07.611291 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:07.611255 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:08.311777 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:08.311739 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:09.497775 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:09.497737 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:11.261056 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:11.260970 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:12.965300 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:12.965260 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:13.142300 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:25:13.142259 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:25:14.712269 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:14.712232 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:15.161313 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:15.161276 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:16.505542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:16.505499 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:18.187923 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:18.187888 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:20.020087 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:20.020047 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:21.641680 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:21.641636 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:21.809758 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:21.809719 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:23.497266 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:23.497229 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:25.237030 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:25.236993 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:27.043851 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:27.043808 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:28.833924 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:28.833889 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:30.568239 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:30.568189 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:32.321104 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:32.321068 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:34.001366 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:34.001326 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:35.735702 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:35.735664 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:36.227252 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:36.227218 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:37.498973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:37.498933 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:39.338388 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:39.338350 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:41.134188 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:41.132707 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk"] Nov 02 08:25:41.236476 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:41.236441 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:41.391106 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:41.383732 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:43.144704 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:25:43.144665 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:25:43.639942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:43.639906 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:45.421676 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:45.421623 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:47.342562 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:47.342526 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:49.102678 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:49.102646 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:50.861369 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:50.861329 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:52.623485 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:52.623447 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:54.411557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:54.411505 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:56.148278 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.148234 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:56.166914 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.166849 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" podUID="991f7ad0-9e4f-40fa-994e-fefbd20f7c09" containerName="main" containerID="cri-o://23c6eec312a770dd4d22e8dccc19d0f0e2fecbacc6c4d205b1f7298a9835843b" gracePeriod=45 Nov 02 08:25:56.307597 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.307553 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" podUID="991f7ad0-9e4f-40fa-994e-fefbd20f7c09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": dial tcp 10.134.0.141:8000: connect: connection refused" Nov 02 08:25:56.442477 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.442450 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:25:56.523487 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.523441 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-model-cache\") pod \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " Nov 02 08:25:56.523681 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.523596 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tls-certs\" (UniqueName: \"kubernetes.io/secret/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tls-certs\") pod \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " Nov 02 08:25:56.524022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.523990 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-model-cache" pod "991f7ad0-9e4f-40fa-994e-fefbd20f7c09" (UID: "991f7ad0-9e4f-40fa-994e-fefbd20f7c09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:25:56.524124 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.524105 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:25:56.526950 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.526925 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tls-certs" pod "991f7ad0-9e4f-40fa-994e-fefbd20f7c09" (UID: "991f7ad0-9e4f-40fa-994e-fefbd20f7c09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:25:56.531501 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.531472 5899 generic.go:356] "Generic (PLEG): container finished" podID="991f7ad0-9e4f-40fa-994e-fefbd20f7c09" containerID="23c6eec312a770dd4d22e8dccc19d0f0e2fecbacc6c4d205b1f7298a9835843b" exitCode=0 Nov 02 08:25:56.531629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.531525 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" Nov 02 08:25:56.531629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.531553 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" event={"ID":"991f7ad0-9e4f-40fa-994e-fefbd20f7c09","Type":"ContainerDied","Data":"23c6eec312a770dd4d22e8dccc19d0f0e2fecbacc6c4d205b1f7298a9835843b"} Nov 02 08:25:56.531629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.531605 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk" event={"ID":"991f7ad0-9e4f-40fa-994e-fefbd20f7c09","Type":"ContainerDied","Data":"8da09d74e9f0bc90711c89391782462af42dad9588257533bdabb7e6d60d9298"} Nov 02 08:25:56.531629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.531629 5899 scope.go:122] "RemoveContainer" containerID="23c6eec312a770dd4d22e8dccc19d0f0e2fecbacc6c4d205b1f7298a9835843b" Nov 02 08:25:56.545608 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.545586 5899 scope.go:122] "RemoveContainer" containerID="23c6eec312a770dd4d22e8dccc19d0f0e2fecbacc6c4d205b1f7298a9835843b" Nov 02 08:25:56.545894 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:25:56.545876 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c6eec312a770dd4d22e8dccc19d0f0e2fecbacc6c4d205b1f7298a9835843b\": container with ID starting with 23c6eec312a770dd4d22e8dccc19d0f0e2fecbacc6c4d205b1f7298a9835843b not found: ID does not exist" containerID="23c6eec312a770dd4d22e8dccc19d0f0e2fecbacc6c4d205b1f7298a9835843b" Nov 02 08:25:56.545948 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.545904 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c6eec312a770dd4d22e8dccc19d0f0e2fecbacc6c4d205b1f7298a9835843b"} err="failed to get container status \"23c6eec312a770dd4d22e8dccc19d0f0e2fecbacc6c4d205b1f7298a9835843b\": rpc error: code = NotFound desc = could not find container \"23c6eec312a770dd4d22e8dccc19d0f0e2fecbacc6c4d205b1f7298a9835843b\": container with ID starting with 23c6eec312a770dd4d22e8dccc19d0f0e2fecbacc6c4d205b1f7298a9835843b not found: ID does not exist" Nov 02 08:25:56.624748 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.624709 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-home\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-home\") pod \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " Nov 02 08:25:56.624936 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.624858 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-dshm\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-dshm\") pod \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " Nov 02 08:25:56.624936 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.624934 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-kube-api-access-r9thf\" (UniqueName: \"kubernetes.io/projected/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-kube-api-access-r9thf\") pod \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " Nov 02 08:25:56.625063 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.624958 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tmp-dir\") pod \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\" (UID: \"991f7ad0-9e4f-40fa-994e-fefbd20f7c09\") " Nov 02 08:25:56.625063 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.624954 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-home" pod "991f7ad0-9e4f-40fa-994e-fefbd20f7c09" (UID: "991f7ad0-9e4f-40fa-994e-fefbd20f7c09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:25:56.625305 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.625281 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tmp-dir" pod "991f7ad0-9e4f-40fa-994e-fefbd20f7c09" (UID: "991f7ad0-9e4f-40fa-994e-fefbd20f7c09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:25:56.625360 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.625343 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:25:56.625360 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.625358 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:25:56.627064 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.627040 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-kube-api-access-r9thf" pod "991f7ad0-9e4f-40fa-994e-fefbd20f7c09" (UID: "991f7ad0-9e4f-40fa-994e-fefbd20f7c09"). InnerVolumeSpecName "kube-api-access-r9thf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:25:56.627203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.627108 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-dshm" pod "991f7ad0-9e4f-40fa-994e-fefbd20f7c09" (UID: "991f7ad0-9e4f-40fa-994e-fefbd20f7c09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:25:56.726922 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.726895 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:25:56.727093 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.726925 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9thf\" (UniqueName: \"kubernetes.io/projected/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-kube-api-access-r9thf\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:25:56.727093 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.726939 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991f7ad0-9e4f-40fa-994e-fefbd20f7c09-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:25:56.879035 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.879000 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk"] Nov 02 08:25:56.884689 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:56.884653 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-78ddbc/test-model-78ddbc-kserve-747b586487-bp4bk"] Nov 02 08:25:57.821841 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:57.821807 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="991f7ad0-9e4f-40fa-994e-fefbd20f7c09" path="/var/lib/kubelet/pods/991f7ad0-9e4f-40fa-994e-fefbd20f7c09/volumes" Nov 02 08:25:57.903053 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:57.903021 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:59.301720 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:59.301686 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:59.819137 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:59.819108 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:25:59.991657 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:25:59.991623 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:01.378795 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:01.378763 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:01.634615 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:01.634521 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:01.835065 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:01.835030 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:03.476994 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:03.476951 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:05.508428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:05.508387 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:06.637642 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:06.637597 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:06.853534 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:06.853488 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:07.216311 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:07.216275 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:07.649761 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:07.649726 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:08.017429 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:08.017391 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:08.347871 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:08.347781 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:08.506101 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:08.506067 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:09.197441 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:09.197404 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:09.709581 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:09.709535 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:11.587839 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:11.587808 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:13.143267 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:26:13.143229 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:26:13.261070 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:13.261032 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:13.405127 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:13.405048 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:14.847456 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:14.847419 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:15.126753 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:15.126669 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:16.971770 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:16.971732 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:18.697796 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:18.697767 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:20.520250 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:20.520213 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:20.617376 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:20.617325 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt"] Nov 02 08:26:20.690883 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:20.690846 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:22.405043 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:22.405006 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:23.873284 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:23.873248 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:24.333324 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:24.333292 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:24.475854 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:24.475824 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:24.718953 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:24.718870 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:26.445351 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:26.445316 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:28.233684 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:28.233648 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:30.030415 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:30.030376 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:30.517742 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:30.517709 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:30.660462 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:30.660427 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:31.288676 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:31.288643 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:31.730001 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:31.729967 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:32.146319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:32.146213 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:32.514385 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:32.514347 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:34.100138 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:34.100100 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:35.511376 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.511338 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs"] Nov 02 08:26:35.511919 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.511890 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" podUID="16310be4-c7d0-44de-bfee-579839e26705" containerName="maas-api" containerID="cri-o://83c2c31a7700180f5119c16d7fa98c6f11d3394d11b3f4d8bbea110cb9d1ec98" gracePeriod=30 Nov 02 08:26:35.626817 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.626783 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:35.643084 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.643036 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" podUID="e766ca8e-466e-4ec7-aacf-758079b07d65" containerName="main" containerID="cri-o://7a6d45c4bac3f703fe8f55e8768d8286ce3e2a6a9e40b33509031b82bab68d45" gracePeriod=45 Nov 02 08:26:35.841680 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.841590 5899 generic.go:356] "Generic (PLEG): container finished" podID="16310be4-c7d0-44de-bfee-579839e26705" containerID="83c2c31a7700180f5119c16d7fa98c6f11d3394d11b3f4d8bbea110cb9d1ec98" exitCode=0 Nov 02 08:26:35.848398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.845250 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:26:35.848398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.845637 5899 generic.go:356] "Generic (PLEG): container finished" podID="e766ca8e-466e-4ec7-aacf-758079b07d65" containerID="7a6d45c4bac3f703fe8f55e8768d8286ce3e2a6a9e40b33509031b82bab68d45" exitCode=0 Nov 02 08:26:35.848398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.848127 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" event={"ID":"16310be4-c7d0-44de-bfee-579839e26705","Type":"ContainerDied","Data":"83c2c31a7700180f5119c16d7fa98c6f11d3394d11b3f4d8bbea110cb9d1ec98"} Nov 02 08:26:35.848398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.848205 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" event={"ID":"16310be4-c7d0-44de-bfee-579839e26705","Type":"ContainerDied","Data":"f82bc473f5bed94871299453ec6a7110118b2b19dde30d6bb5cf9a20b26aed89"} Nov 02 08:26:35.848398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.848235 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" event={"ID":"e766ca8e-466e-4ec7-aacf-758079b07d65","Type":"ContainerDied","Data":"7a6d45c4bac3f703fe8f55e8768d8286ce3e2a6a9e40b33509031b82bab68d45"} Nov 02 08:26:35.848398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.848261 5899 scope.go:122] "RemoveContainer" containerID="83c2c31a7700180f5119c16d7fa98c6f11d3394d11b3f4d8bbea110cb9d1ec98" Nov 02 08:26:35.862207 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.862178 5899 scope.go:122] "RemoveContainer" containerID="83c2c31a7700180f5119c16d7fa98c6f11d3394d11b3f4d8bbea110cb9d1ec98" Nov 02 08:26:35.862599 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:26:35.862576 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c2c31a7700180f5119c16d7fa98c6f11d3394d11b3f4d8bbea110cb9d1ec98\": container with ID starting with 83c2c31a7700180f5119c16d7fa98c6f11d3394d11b3f4d8bbea110cb9d1ec98 not found: ID does not exist" containerID="83c2c31a7700180f5119c16d7fa98c6f11d3394d11b3f4d8bbea110cb9d1ec98" Nov 02 08:26:35.862717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.862611 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c2c31a7700180f5119c16d7fa98c6f11d3394d11b3f4d8bbea110cb9d1ec98"} err="failed to get container status \"83c2c31a7700180f5119c16d7fa98c6f11d3394d11b3f4d8bbea110cb9d1ec98\": rpc error: code = NotFound desc = could not find container \"83c2c31a7700180f5119c16d7fa98c6f11d3394d11b3f4d8bbea110cb9d1ec98\": container with ID starting with 83c2c31a7700180f5119c16d7fa98c6f11d3394d11b3f4d8bbea110cb9d1ec98 not found: ID does not exist" Nov 02 08:26:35.950834 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.950792 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16310be4-c7d0-44de-bfee-579839e26705-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16310be4-c7d0-44de-bfee-579839e26705-openshift-service-ca\") pod \"16310be4-c7d0-44de-bfee-579839e26705\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " Nov 02 08:26:35.951006 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.950845 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-maas-api-tls\") pod \"16310be4-c7d0-44de-bfee-579839e26705\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " Nov 02 08:26:35.951006 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.950884 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16310be4-c7d0-44de-bfee-579839e26705-kube-api-access-78dm8\" (UniqueName: \"kubernetes.io/projected/16310be4-c7d0-44de-bfee-579839e26705-kube-api-access-78dm8\") pod \"16310be4-c7d0-44de-bfee-579839e26705\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " Nov 02 08:26:35.951128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.951086 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-metrics-certs\") pod \"16310be4-c7d0-44de-bfee-579839e26705\" (UID: \"16310be4-c7d0-44de-bfee-579839e26705\") " Nov 02 08:26:35.951245 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.951216 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16310be4-c7d0-44de-bfee-579839e26705-openshift-service-ca" pod "16310be4-c7d0-44de-bfee-579839e26705" (UID: "16310be4-c7d0-44de-bfee-579839e26705"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:26:35.952076 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.952046 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16310be4-c7d0-44de-bfee-579839e26705-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:35.953345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.953316 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-maas-api-tls" pod "16310be4-c7d0-44de-bfee-579839e26705" (UID: "16310be4-c7d0-44de-bfee-579839e26705"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:26:35.953345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.953337 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-metrics-certs" pod "16310be4-c7d0-44de-bfee-579839e26705" (UID: "16310be4-c7d0-44de-bfee-579839e26705"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:26:35.953497 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.953326 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16310be4-c7d0-44de-bfee-579839e26705-kube-api-access-78dm8" pod "16310be4-c7d0-44de-bfee-579839e26705" (UID: "16310be4-c7d0-44de-bfee-579839e26705"). InnerVolumeSpecName "kube-api-access-78dm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:26:35.956221 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:35.956195 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:26:36.053355 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.053258 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-model-cache\") pod \"e766ca8e-466e-4ec7-aacf-758079b07d65\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " Nov 02 08:26:36.053659 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.053633 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-dshm\") pod \"e766ca8e-466e-4ec7-aacf-758079b07d65\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " Nov 02 08:26:36.054416 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.054392 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e766ca8e-466e-4ec7-aacf-758079b07d65-tls-certs\" (UniqueName: \"kubernetes.io/secret/e766ca8e-466e-4ec7-aacf-758079b07d65-tls-certs\") pod \"e766ca8e-466e-4ec7-aacf-758079b07d65\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " Nov 02 08:26:36.054688 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.054655 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-tmp-dir\") pod \"e766ca8e-466e-4ec7-aacf-758079b07d65\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " Nov 02 08:26:36.054875 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.054858 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e766ca8e-466e-4ec7-aacf-758079b07d65-kube-api-access-sbvhs\" (UniqueName: \"kubernetes.io/projected/e766ca8e-466e-4ec7-aacf-758079b07d65-kube-api-access-sbvhs\") pod \"e766ca8e-466e-4ec7-aacf-758079b07d65\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " Nov 02 08:26:36.055091 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.055059 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-home\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-home\") pod \"e766ca8e-466e-4ec7-aacf-758079b07d65\" (UID: \"e766ca8e-466e-4ec7-aacf-758079b07d65\") " Nov 02 08:26:36.055994 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.055973 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:36.056193 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.056150 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78dm8\" (UniqueName: \"kubernetes.io/projected/16310be4-c7d0-44de-bfee-579839e26705-kube-api-access-78dm8\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:36.056349 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.056318 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16310be4-c7d0-44de-bfee-579839e26705-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:36.057208 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.053594 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-model-cache" pod "e766ca8e-466e-4ec7-aacf-758079b07d65" (UID: "e766ca8e-466e-4ec7-aacf-758079b07d65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:36.057350 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.056733 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-home" pod "e766ca8e-466e-4ec7-aacf-758079b07d65" (UID: "e766ca8e-466e-4ec7-aacf-758079b07d65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:36.057524 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.056925 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-tmp-dir" pod "e766ca8e-466e-4ec7-aacf-758079b07d65" (UID: "e766ca8e-466e-4ec7-aacf-758079b07d65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:36.059232 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.059199 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e766ca8e-466e-4ec7-aacf-758079b07d65-tls-certs" pod "e766ca8e-466e-4ec7-aacf-758079b07d65" (UID: "e766ca8e-466e-4ec7-aacf-758079b07d65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:26:36.059232 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.059216 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-dshm" pod "e766ca8e-466e-4ec7-aacf-758079b07d65" (UID: "e766ca8e-466e-4ec7-aacf-758079b07d65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:36.059609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.059581 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e766ca8e-466e-4ec7-aacf-758079b07d65-kube-api-access-sbvhs" pod "e766ca8e-466e-4ec7-aacf-758079b07d65" (UID: "e766ca8e-466e-4ec7-aacf-758079b07d65"). InnerVolumeSpecName "kube-api-access-sbvhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:26:36.158054 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.158019 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:36.158054 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.158053 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:36.158312 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.158068 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e766ca8e-466e-4ec7-aacf-758079b07d65-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:36.158312 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.158079 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:36.158312 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.158093 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbvhs\" (UniqueName: \"kubernetes.io/projected/e766ca8e-466e-4ec7-aacf-758079b07d65-kube-api-access-sbvhs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:36.158312 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.158107 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e766ca8e-466e-4ec7-aacf-758079b07d65-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:36.434263 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.434156 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:36.853188 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.853145 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs" Nov 02 08:26:36.854789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.854755 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" Nov 02 08:26:36.854931 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.854765 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt" event={"ID":"e766ca8e-466e-4ec7-aacf-758079b07d65","Type":"ContainerDied","Data":"0d4c22cf7ca3281913e611720f21aba5066bac9973b0c54c2a39cb2891e7ebce"} Nov 02 08:26:36.854931 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.854889 5899 scope.go:122] "RemoveContainer" containerID="7a6d45c4bac3f703fe8f55e8768d8286ce3e2a6a9e40b33509031b82bab68d45" Nov 02 08:26:36.907921 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.907885 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs"] Nov 02 08:26:36.920798 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.920768 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e6f4fe-7697946c7f-bkjvs"] Nov 02 08:26:36.948136 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.948100 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt"] Nov 02 08:26:36.955215 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:36.955179 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b72fa3/test-model-b72fa3-kserve-66947968fc-2snrt"] Nov 02 08:26:37.676561 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:37.676477 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:37.819726 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:37.819691 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:37.822628 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:37.822602 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16310be4-c7d0-44de-bfee-579839e26705" path="/var/lib/kubelet/pods/16310be4-c7d0-44de-bfee-579839e26705/volumes" Nov 02 08:26:37.823093 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:37.823079 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e766ca8e-466e-4ec7-aacf-758079b07d65" path="/var/lib/kubelet/pods/e766ca8e-466e-4ec7-aacf-758079b07d65/volumes" Nov 02 08:26:38.449466 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:38.449433 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:39.037553 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:39.037524 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:39.194577 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:39.194542 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:39.874471 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:39.874437 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:39.892345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:39.892304 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9"] Nov 02 08:26:40.053839 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:40.053803 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:40.212337 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:40.212256 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:40.461587 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:40.461556 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:40.672100 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:40.672024 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:40.865494 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:40.865445 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:41.027578 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:41.027542 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:42.702052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:42.702018 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:43.141999 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:26:43.141966 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:26:44.450432 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:44.450397 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:45.692559 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:45.692521 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:45.886677 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:45.886643 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:46.090311 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:46.090279 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:46.398398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:46.398316 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:46.997620 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:46.997575 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:48.330131 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:48.330099 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:48.500962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:48.500929 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:48.683363 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:48.683282 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:48.946383 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:48.946272 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:50.397090 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:50.397048 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:52.031316 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:52.031275 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:53.752960 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:53.752926 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:54.922029 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:54.921987 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" podUID="1a92d409-9023-4994-9c49-86bed431a47a" containerName="main" containerID="cri-o://e1c5546360982d8e5d41892156a02bce2750a007d9ba7b5b114fb491388e91d9" gracePeriod=45 Nov 02 08:26:55.001904 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.001869 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:55.143467 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.143436 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:55.206452 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.206424 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:26:55.309542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.309504 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a92d409-9023-4994-9c49-86bed431a47a-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a92d409-9023-4994-9c49-86bed431a47a-tls-certs\") pod \"1a92d409-9023-4994-9c49-86bed431a47a\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " Nov 02 08:26:55.309697 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.309625 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-home\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-home\") pod \"1a92d409-9023-4994-9c49-86bed431a47a\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " Nov 02 08:26:55.309697 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.309674 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a92d409-9023-4994-9c49-86bed431a47a-kube-api-access-xqmrm\" (UniqueName: \"kubernetes.io/projected/1a92d409-9023-4994-9c49-86bed431a47a-kube-api-access-xqmrm\") pod \"1a92d409-9023-4994-9c49-86bed431a47a\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " Nov 02 08:26:55.309819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.309710 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-tmp-dir\") pod \"1a92d409-9023-4994-9c49-86bed431a47a\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " Nov 02 08:26:55.309819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.309777 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-model-cache\") pod \"1a92d409-9023-4994-9c49-86bed431a47a\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " Nov 02 08:26:55.309961 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.309926 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-dshm\") pod \"1a92d409-9023-4994-9c49-86bed431a47a\" (UID: \"1a92d409-9023-4994-9c49-86bed431a47a\") " Nov 02 08:26:55.310022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.309962 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-home" pod "1a92d409-9023-4994-9c49-86bed431a47a" (UID: "1a92d409-9023-4994-9c49-86bed431a47a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:55.310022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.310001 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-tmp-dir" pod "1a92d409-9023-4994-9c49-86bed431a47a" (UID: "1a92d409-9023-4994-9c49-86bed431a47a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:55.310139 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.310088 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-model-cache" pod "1a92d409-9023-4994-9c49-86bed431a47a" (UID: "1a92d409-9023-4994-9c49-86bed431a47a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:55.310440 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.310419 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:55.310440 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.310441 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:55.310605 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.310450 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:55.311942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.311914 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a92d409-9023-4994-9c49-86bed431a47a-tls-certs" pod "1a92d409-9023-4994-9c49-86bed431a47a" (UID: "1a92d409-9023-4994-9c49-86bed431a47a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:26:55.312036 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.311936 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-dshm" pod "1a92d409-9023-4994-9c49-86bed431a47a" (UID: "1a92d409-9023-4994-9c49-86bed431a47a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:26:55.312036 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.311940 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a92d409-9023-4994-9c49-86bed431a47a-kube-api-access-xqmrm" pod "1a92d409-9023-4994-9c49-86bed431a47a" (UID: "1a92d409-9023-4994-9c49-86bed431a47a"). InnerVolumeSpecName "kube-api-access-xqmrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:26:55.411667 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.411632 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a92d409-9023-4994-9c49-86bed431a47a-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:55.411667 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.411661 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a92d409-9023-4994-9c49-86bed431a47a-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:55.411667 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.411670 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqmrm\" (UniqueName: \"kubernetes.io/projected/1a92d409-9023-4994-9c49-86bed431a47a-kube-api-access-xqmrm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:26:55.511986 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.511943 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:55.993647 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.993611 5899 generic.go:356] "Generic (PLEG): container finished" podID="1a92d409-9023-4994-9c49-86bed431a47a" containerID="e1c5546360982d8e5d41892156a02bce2750a007d9ba7b5b114fb491388e91d9" exitCode=0 Nov 02 08:26:55.994082 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.993663 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" Nov 02 08:26:55.994082 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.993699 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" event={"ID":"1a92d409-9023-4994-9c49-86bed431a47a","Type":"ContainerDied","Data":"e1c5546360982d8e5d41892156a02bce2750a007d9ba7b5b114fb491388e91d9"} Nov 02 08:26:55.994082 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.993742 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9" event={"ID":"1a92d409-9023-4994-9c49-86bed431a47a","Type":"ContainerDied","Data":"04e543611873d6723c77cd75b83790a0a2b7876941e7fc3381e275d43ffd0024"} Nov 02 08:26:55.994082 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:55.993759 5899 scope.go:122] "RemoveContainer" containerID="e1c5546360982d8e5d41892156a02bce2750a007d9ba7b5b114fb491388e91d9" Nov 02 08:26:56.005913 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:56.005885 5899 scope.go:122] "RemoveContainer" containerID="e1c5546360982d8e5d41892156a02bce2750a007d9ba7b5b114fb491388e91d9" Nov 02 08:26:56.006254 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:26:56.006227 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c5546360982d8e5d41892156a02bce2750a007d9ba7b5b114fb491388e91d9\": container with ID starting with e1c5546360982d8e5d41892156a02bce2750a007d9ba7b5b114fb491388e91d9 not found: ID does not exist" containerID="e1c5546360982d8e5d41892156a02bce2750a007d9ba7b5b114fb491388e91d9" Nov 02 08:26:56.006376 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:56.006263 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c5546360982d8e5d41892156a02bce2750a007d9ba7b5b114fb491388e91d9"} err="failed to get container status \"e1c5546360982d8e5d41892156a02bce2750a007d9ba7b5b114fb491388e91d9\": rpc error: code = NotFound desc = could not find container \"e1c5546360982d8e5d41892156a02bce2750a007d9ba7b5b114fb491388e91d9\": container with ID starting with e1c5546360982d8e5d41892156a02bce2750a007d9ba7b5b114fb491388e91d9 not found: ID does not exist" Nov 02 08:26:56.048641 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:56.048602 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9"] Nov 02 08:26:56.061644 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:56.061610 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-eb6010/test-model-eb6010-kserve-869dc6f78c-82wm9"] Nov 02 08:26:56.449894 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:56.449857 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:57.210809 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:57.210775 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:57.419195 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:57.419122 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:26:57.821676 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:57.821640 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a92d409-9023-4994-9c49-86bed431a47a" path="/var/lib/kubelet/pods/1a92d409-9023-4994-9c49-86bed431a47a/volumes" Nov 02 08:26:59.073118 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:26:59.073084 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:00.897353 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:00.897312 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:02.645487 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:02.645446 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:04.189867 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:04.189822 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:04.405365 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:04.405328 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:06.075542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:06.075502 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:07.747723 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:07.747683 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:08.610554 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:08.610517 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:09.619360 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:09.619325 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:11.392064 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:11.392025 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:13.141754 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:27:13.141719 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:27:13.172212 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:13.172142 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:14.897501 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:14.897462 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:16.707094 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:16.707060 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:18.411396 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:18.411357 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:20.170601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:20.170562 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:21.662359 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:21.662322 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8"] Nov 02 08:27:21.662914 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:21.662580 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" podUID="e18ba88c-ba9d-4ab4-be5e-284c239a9228" containerName="maas-api" containerID="cri-o://014b973fedd488a6a1fcfb4b9410bd5f2c5a537fd339e57436d876a20ea048af" gracePeriod=30 Nov 02 08:27:21.759861 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:21.759813 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:21.969552 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:21.969523 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:27:22.057109 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.057072 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e18ba88c-ba9d-4ab4-be5e-284c239a9228-kube-api-access-mrc4m\" (UniqueName: \"kubernetes.io/projected/e18ba88c-ba9d-4ab4-be5e-284c239a9228-kube-api-access-mrc4m\") pod \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " Nov 02 08:27:22.057335 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.057126 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e18ba88c-ba9d-4ab4-be5e-284c239a9228-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18ba88c-ba9d-4ab4-be5e-284c239a9228-openshift-service-ca\") pod \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " Nov 02 08:27:22.057335 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.057267 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-metrics-certs\") pod \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " Nov 02 08:27:22.057475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.057386 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-maas-api-tls\") pod \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\" (UID: \"e18ba88c-ba9d-4ab4-be5e-284c239a9228\") " Nov 02 08:27:22.057572 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.057544 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18ba88c-ba9d-4ab4-be5e-284c239a9228-openshift-service-ca" pod "e18ba88c-ba9d-4ab4-be5e-284c239a9228" (UID: "e18ba88c-ba9d-4ab4-be5e-284c239a9228"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:27:22.057989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.057961 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18ba88c-ba9d-4ab4-be5e-284c239a9228-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:27:22.059405 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.059376 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-metrics-certs" pod "e18ba88c-ba9d-4ab4-be5e-284c239a9228" (UID: "e18ba88c-ba9d-4ab4-be5e-284c239a9228"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:27:22.059506 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.059487 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e18ba88c-ba9d-4ab4-be5e-284c239a9228-kube-api-access-mrc4m" pod "e18ba88c-ba9d-4ab4-be5e-284c239a9228" (UID: "e18ba88c-ba9d-4ab4-be5e-284c239a9228"). InnerVolumeSpecName "kube-api-access-mrc4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:27:22.059722 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.059691 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-maas-api-tls" pod "e18ba88c-ba9d-4ab4-be5e-284c239a9228" (UID: "e18ba88c-ba9d-4ab4-be5e-284c239a9228"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:27:22.159408 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.159378 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:27:22.159408 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.159404 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18ba88c-ba9d-4ab4-be5e-284c239a9228-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:27:22.159408 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.159415 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrc4m\" (UniqueName: \"kubernetes.io/projected/e18ba88c-ba9d-4ab4-be5e-284c239a9228-kube-api-access-mrc4m\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:27:22.161768 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.161740 5899 generic.go:356] "Generic (PLEG): container finished" podID="e18ba88c-ba9d-4ab4-be5e-284c239a9228" containerID="014b973fedd488a6a1fcfb4b9410bd5f2c5a537fd339e57436d876a20ea048af" exitCode=0 Nov 02 08:27:22.161906 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.161774 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" event={"ID":"e18ba88c-ba9d-4ab4-be5e-284c239a9228","Type":"ContainerDied","Data":"014b973fedd488a6a1fcfb4b9410bd5f2c5a537fd339e57436d876a20ea048af"} Nov 02 08:27:22.161906 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.161788 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" Nov 02 08:27:22.161906 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.161812 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8" event={"ID":"e18ba88c-ba9d-4ab4-be5e-284c239a9228","Type":"ContainerDied","Data":"ac97a135bca4ef6e675316cd1b41aa0cd7bc26e14bfa798c560850571f1a2e7d"} Nov 02 08:27:22.161906 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.161834 5899 scope.go:122] "RemoveContainer" containerID="014b973fedd488a6a1fcfb4b9410bd5f2c5a537fd339e57436d876a20ea048af" Nov 02 08:27:22.173553 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.173532 5899 scope.go:122] "RemoveContainer" containerID="014b973fedd488a6a1fcfb4b9410bd5f2c5a537fd339e57436d876a20ea048af" Nov 02 08:27:22.173826 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:27:22.173805 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014b973fedd488a6a1fcfb4b9410bd5f2c5a537fd339e57436d876a20ea048af\": container with ID starting with 014b973fedd488a6a1fcfb4b9410bd5f2c5a537fd339e57436d876a20ea048af not found: ID does not exist" containerID="014b973fedd488a6a1fcfb4b9410bd5f2c5a537fd339e57436d876a20ea048af" Nov 02 08:27:22.173889 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.173835 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014b973fedd488a6a1fcfb4b9410bd5f2c5a537fd339e57436d876a20ea048af"} err="failed to get container status \"014b973fedd488a6a1fcfb4b9410bd5f2c5a537fd339e57436d876a20ea048af\": rpc error: code = NotFound desc = could not find container \"014b973fedd488a6a1fcfb4b9410bd5f2c5a537fd339e57436d876a20ea048af\": container with ID starting with 014b973fedd488a6a1fcfb4b9410bd5f2c5a537fd339e57436d876a20ea048af not found: ID does not exist" Nov 02 08:27:22.219664 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.219619 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8"] Nov 02 08:27:22.225349 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.225313 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d042a8-55584968c9-7mgx8"] Nov 02 08:27:22.560782 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:22.560746 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:23.820557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:23.820523 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e18ba88c-ba9d-4ab4-be5e-284c239a9228" path="/var/lib/kubelet/pods/e18ba88c-ba9d-4ab4-be5e-284c239a9228/volumes" Nov 02 08:27:24.273758 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:24.273730 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:26.038143 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:26.038099 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:27.512513 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:27.512461 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:28.289279 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:28.289242 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:30.095869 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:30.095828 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:30.610548 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:30.610517 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:30.750820 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:30.750786 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:31.335574 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:31.335534 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:31.479831 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:31.479797 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:31.539587 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:31.539552 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9"] Nov 02 08:27:31.539779 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:31.539755 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" podUID="71135335-1f07-4594-b522-1a79379b31b6" containerName="maas-api" containerID="cri-o://841f120ab226b31567f4b5d34bd7c3c71c004a927518491f26c5355d951b94ed" gracePeriod=30 Nov 02 08:27:31.676187 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:31.671650 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:31.812794 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:31.812720 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:31.925062 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:31.924984 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:27:32.011196 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.010894 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-metrics-certs\") pod \"71135335-1f07-4594-b522-1a79379b31b6\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " Nov 02 08:27:32.011196 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.010990 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71135335-1f07-4594-b522-1a79379b31b6-kube-api-access-9b5bv\" (UniqueName: \"kubernetes.io/projected/71135335-1f07-4594-b522-1a79379b31b6-kube-api-access-9b5bv\") pod \"71135335-1f07-4594-b522-1a79379b31b6\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " Nov 02 08:27:32.011196 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.011113 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-maas-api-tls\") pod \"71135335-1f07-4594-b522-1a79379b31b6\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " Nov 02 08:27:32.011196 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.011147 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71135335-1f07-4594-b522-1a79379b31b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71135335-1f07-4594-b522-1a79379b31b6-openshift-service-ca\") pod \"71135335-1f07-4594-b522-1a79379b31b6\" (UID: \"71135335-1f07-4594-b522-1a79379b31b6\") " Nov 02 08:27:32.024800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.024751 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71135335-1f07-4594-b522-1a79379b31b6-openshift-service-ca" pod "71135335-1f07-4594-b522-1a79379b31b6" (UID: "71135335-1f07-4594-b522-1a79379b31b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:27:32.027386 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.027354 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-maas-api-tls" pod "71135335-1f07-4594-b522-1a79379b31b6" (UID: "71135335-1f07-4594-b522-1a79379b31b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:27:32.033528 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.033500 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71135335-1f07-4594-b522-1a79379b31b6-kube-api-access-9b5bv" pod "71135335-1f07-4594-b522-1a79379b31b6" (UID: "71135335-1f07-4594-b522-1a79379b31b6"). InnerVolumeSpecName "kube-api-access-9b5bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:27:32.035804 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.035781 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-metrics-certs" pod "71135335-1f07-4594-b522-1a79379b31b6" (UID: "71135335-1f07-4594-b522-1a79379b31b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:27:32.114076 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.114036 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:27:32.114076 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.114079 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b5bv\" (UniqueName: \"kubernetes.io/projected/71135335-1f07-4594-b522-1a79379b31b6-kube-api-access-9b5bv\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:27:32.114344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.114094 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71135335-1f07-4594-b522-1a79379b31b6-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:27:32.114344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.114114 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71135335-1f07-4594-b522-1a79379b31b6-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:27:32.220460 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.220367 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:32.230194 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.230151 5899 generic.go:356] "Generic (PLEG): container finished" podID="71135335-1f07-4594-b522-1a79379b31b6" containerID="841f120ab226b31567f4b5d34bd7c3c71c004a927518491f26c5355d951b94ed" exitCode=0 Nov 02 08:27:32.230194 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.230185 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" event={"ID":"71135335-1f07-4594-b522-1a79379b31b6","Type":"ContainerDied","Data":"841f120ab226b31567f4b5d34bd7c3c71c004a927518491f26c5355d951b94ed"} Nov 02 08:27:32.230386 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.230227 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" Nov 02 08:27:32.230386 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.230265 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9" event={"ID":"71135335-1f07-4594-b522-1a79379b31b6","Type":"ContainerDied","Data":"83362aac7206f2c288c04bb298e6eea87166cbdb52568b15d912ea4c61078ca8"} Nov 02 08:27:32.230386 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.230288 5899 scope.go:122] "RemoveContainer" containerID="841f120ab226b31567f4b5d34bd7c3c71c004a927518491f26c5355d951b94ed" Nov 02 08:27:32.241302 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.241281 5899 scope.go:122] "RemoveContainer" containerID="841f120ab226b31567f4b5d34bd7c3c71c004a927518491f26c5355d951b94ed" Nov 02 08:27:32.241575 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:27:32.241555 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841f120ab226b31567f4b5d34bd7c3c71c004a927518491f26c5355d951b94ed\": container with ID starting with 841f120ab226b31567f4b5d34bd7c3c71c004a927518491f26c5355d951b94ed not found: ID does not exist" containerID="841f120ab226b31567f4b5d34bd7c3c71c004a927518491f26c5355d951b94ed" Nov 02 08:27:32.241630 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.241584 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841f120ab226b31567f4b5d34bd7c3c71c004a927518491f26c5355d951b94ed"} err="failed to get container status \"841f120ab226b31567f4b5d34bd7c3c71c004a927518491f26c5355d951b94ed\": rpc error: code = NotFound desc = could not find container \"841f120ab226b31567f4b5d34bd7c3c71c004a927518491f26c5355d951b94ed\": container with ID starting with 841f120ab226b31567f4b5d34bd7c3c71c004a927518491f26c5355d951b94ed not found: ID does not exist" Nov 02 08:27:32.284761 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.284722 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9"] Nov 02 08:27:32.297806 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.297773 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ba44ed-75d9ffdb5b-9jxj9"] Nov 02 08:27:32.406836 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.406798 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:32.551150 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.551115 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:32.693512 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:32.693474 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:33.705827 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:33.705776 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:33.820911 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:33.820873 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71135335-1f07-4594-b522-1a79379b31b6" path="/var/lib/kubelet/pods/71135335-1f07-4594-b522-1a79379b31b6/volumes" Nov 02 08:27:34.143826 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:34.143785 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:34.277458 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:34.277423 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:34.427991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:34.427904 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:36.127725 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:36.127694 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:37.884896 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:37.884866 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:38.033191 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:38.033125 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:38.242296 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:38.242268 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:38.437899 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:38.437860 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:40.068280 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:40.068242 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:40.275427 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:40.275383 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:40.447669 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:40.447586 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:40.603243 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:40.603200 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:40.754201 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:40.754155 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:41.341491 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:41.341454 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:41.482307 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:41.482269 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:41.775705 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:41.775668 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:42.376703 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:42.376669 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:42.780024 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:42.779994 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:42.935499 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:42.935465 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:43.079660 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:43.079579 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:43.140711 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:27:43.140670 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:27:44.059028 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:44.058989 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:44.249156 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:44.249117 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:44.387723 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:44.387635 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:46.039014 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:46.038980 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:47.589228 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:47.589188 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:47.798514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:47.798474 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:49.621116 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:49.621083 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:51.351101 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:51.351068 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:53.224490 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:53.224454 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:55.056848 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:55.056817 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:56.260304 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:56.260261 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:56.730970 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:56.730940 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:27:58.510370 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:27:58.510331 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:00.307069 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:00.307030 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:02.042613 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:02.042572 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:02.173128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:02.173091 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:03.731426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:03.731390 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:05.419477 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:05.419444 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:07.241462 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:07.241419 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:08.695966 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:08.695928 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:08.999987 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:08.999952 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:10.793235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:10.789618 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:10.934442 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:10.934404 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:11.212242 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:11.208246 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:13.141265 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:28:13.141228 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:28:13.442062 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:13.441967 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:15.123650 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:15.123607 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:16.916910 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:16.916876 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:17.242716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:17.242671 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:17.407212 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:17.407144 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:18.455506 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.455471 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz"] Nov 02 08:28:18.455916 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.455709 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" podUID="a1880d50-8522-44ed-bfff-a4567776155c" containerName="maas-api" containerID="cri-o://5a820622339b516eaf601287ccc0f97268870d3bded32ef5b3df6c712948837f" gracePeriod=30 Nov 02 08:28:18.577721 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.577675 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:18.731140 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.731107 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:28:18.820900 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.820853 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-metrics-certs\") pod \"a1880d50-8522-44ed-bfff-a4567776155c\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " Nov 02 08:28:18.821117 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.820940 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1880d50-8522-44ed-bfff-a4567776155c-kube-api-access-ln8w7\" (UniqueName: \"kubernetes.io/projected/a1880d50-8522-44ed-bfff-a4567776155c-kube-api-access-ln8w7\") pod \"a1880d50-8522-44ed-bfff-a4567776155c\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " Nov 02 08:28:18.821117 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.821043 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-maas-api-tls\") pod \"a1880d50-8522-44ed-bfff-a4567776155c\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " Nov 02 08:28:18.821314 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.821136 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1880d50-8522-44ed-bfff-a4567776155c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1880d50-8522-44ed-bfff-a4567776155c-openshift-service-ca\") pod \"a1880d50-8522-44ed-bfff-a4567776155c\" (UID: \"a1880d50-8522-44ed-bfff-a4567776155c\") " Nov 02 08:28:18.822235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.822205 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1880d50-8522-44ed-bfff-a4567776155c-openshift-service-ca" pod "a1880d50-8522-44ed-bfff-a4567776155c" (UID: "a1880d50-8522-44ed-bfff-a4567776155c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:28:18.825089 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.825061 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1880d50-8522-44ed-bfff-a4567776155c-kube-api-access-ln8w7" pod "a1880d50-8522-44ed-bfff-a4567776155c" (UID: "a1880d50-8522-44ed-bfff-a4567776155c"). InnerVolumeSpecName "kube-api-access-ln8w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:28:18.825387 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.825353 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-maas-api-tls" pod "a1880d50-8522-44ed-bfff-a4567776155c" (UID: "a1880d50-8522-44ed-bfff-a4567776155c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:28:18.833338 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.833290 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-metrics-certs" pod "a1880d50-8522-44ed-bfff-a4567776155c" (UID: "a1880d50-8522-44ed-bfff-a4567776155c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:28:18.923647 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.923610 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:28:18.923647 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.923647 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1880d50-8522-44ed-bfff-a4567776155c-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:28:18.923860 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.923665 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1880d50-8522-44ed-bfff-a4567776155c-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:28:18.923860 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:18.923685 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln8w7\" (UniqueName: \"kubernetes.io/projected/a1880d50-8522-44ed-bfff-a4567776155c-kube-api-access-ln8w7\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:28:19.282475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.282437 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:19.306279 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306245 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775"] Nov 02 08:28:19.306843 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306828 5899 cpu_manager.go:414] "containerMap: removing container" podUID="991f7ad0-9e4f-40fa-994e-fefbd20f7c09" containerName="main" Nov 02 08:28:19.306890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306844 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991f7ad0-9e4f-40fa-994e-fefbd20f7c09" containerName="main" Nov 02 08:28:19.306890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306879 5899 cpu_manager.go:414] "containerMap: removing container" podUID="16310be4-c7d0-44de-bfee-579839e26705" containerName="maas-api" Nov 02 08:28:19.306890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306883 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16310be4-c7d0-44de-bfee-579839e26705" containerName="maas-api" Nov 02 08:28:19.307000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306899 5899 cpu_manager.go:414] "containerMap: removing container" podUID="1a92d409-9023-4994-9c49-86bed431a47a" containerName="main" Nov 02 08:28:19.307000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306904 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a92d409-9023-4994-9c49-86bed431a47a" containerName="main" Nov 02 08:28:19.307000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306913 5899 cpu_manager.go:414] "containerMap: removing container" podUID="e18ba88c-ba9d-4ab4-be5e-284c239a9228" containerName="maas-api" Nov 02 08:28:19.307000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306919 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e18ba88c-ba9d-4ab4-be5e-284c239a9228" containerName="maas-api" Nov 02 08:28:19.307000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306932 5899 cpu_manager.go:414] "containerMap: removing container" podUID="71135335-1f07-4594-b522-1a79379b31b6" containerName="maas-api" Nov 02 08:28:19.307000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306939 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71135335-1f07-4594-b522-1a79379b31b6" containerName="maas-api" Nov 02 08:28:19.307000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306946 5899 cpu_manager.go:414] "containerMap: removing container" podUID="a1880d50-8522-44ed-bfff-a4567776155c" containerName="maas-api" Nov 02 08:28:19.307000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306951 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1880d50-8522-44ed-bfff-a4567776155c" containerName="maas-api" Nov 02 08:28:19.307000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306964 5899 cpu_manager.go:414] "containerMap: removing container" podUID="e766ca8e-466e-4ec7-aacf-758079b07d65" containerName="main" Nov 02 08:28:19.307000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.306968 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e766ca8e-466e-4ec7-aacf-758079b07d65" containerName="main" Nov 02 08:28:19.307333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.307052 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="e766ca8e-466e-4ec7-aacf-758079b07d65" containerName="main" Nov 02 08:28:19.307333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.307063 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="991f7ad0-9e4f-40fa-994e-fefbd20f7c09" containerName="main" Nov 02 08:28:19.307333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.307074 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="16310be4-c7d0-44de-bfee-579839e26705" containerName="maas-api" Nov 02 08:28:19.307333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.307083 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a92d409-9023-4994-9c49-86bed431a47a" containerName="main" Nov 02 08:28:19.307333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.307088 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="e18ba88c-ba9d-4ab4-be5e-284c239a9228" containerName="maas-api" Nov 02 08:28:19.307333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.307094 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="71135335-1f07-4594-b522-1a79379b31b6" containerName="maas-api" Nov 02 08:28:19.307333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.307099 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1880d50-8522-44ed-bfff-a4567776155c" containerName="maas-api" Nov 02 08:28:19.311035 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.311015 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.331782 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.331753 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775"] Nov 02 08:28:19.340848 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.340778 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9877ce\"/\"default-dockercfg-858ql\"" Nov 02 08:28:19.340848 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.340827 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9877ce\"/\"sub-test-model-9877ce-kserve-self-signed-certs\"" Nov 02 08:28:19.413112 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.413075 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9877ce\"/\"kube-root-ca.crt\"" Nov 02 08:28:19.413325 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.413204 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9877ce\"/\"openshift-service-ca.crt\"" Nov 02 08:28:19.427837 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.427772 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:19.432508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.432479 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-home\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.432676 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.432540 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-model-cache\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.432676 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.432631 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwz9z\" (UniqueName: \"kubernetes.io/projected/053fdd58-57a4-42a5-9cd6-4891d05da186-kube-api-access-gwz9z\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.432796 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.432697 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/053fdd58-57a4-42a5-9cd6-4891d05da186-tls-certs\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.432890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.432864 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-tmp-dir\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.432951 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.432906 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-dshm\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.534112 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.534018 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/053fdd58-57a4-42a5-9cd6-4891d05da186-tls-certs\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.534112 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.534111 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-tmp-dir\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.534693 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.534137 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-dshm\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.534693 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.534241 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-home\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.534693 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.534288 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-model-cache\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.534693 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.534348 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwz9z\" (UniqueName: \"kubernetes.io/projected/053fdd58-57a4-42a5-9cd6-4891d05da186-kube-api-access-gwz9z\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.534693 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.534638 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-home\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.534990 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.534704 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-model-cache\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.534990 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.534940 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-tmp-dir\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.537040 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.537006 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-dshm\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.537421 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.537398 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/053fdd58-57a4-42a5-9cd6-4891d05da186-tls-certs\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.559584 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.559517 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwz9z\" (UniqueName: \"kubernetes.io/projected/053fdd58-57a4-42a5-9cd6-4891d05da186-kube-api-access-gwz9z\") pod \"sub-test-model-9877ce-kserve-649776c967-h6775\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.579322 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.579282 5899 generic.go:356] "Generic (PLEG): container finished" podID="a1880d50-8522-44ed-bfff-a4567776155c" containerID="5a820622339b516eaf601287ccc0f97268870d3bded32ef5b3df6c712948837f" exitCode=0 Nov 02 08:28:19.579524 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.579391 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" event={"ID":"a1880d50-8522-44ed-bfff-a4567776155c","Type":"ContainerDied","Data":"5a820622339b516eaf601287ccc0f97268870d3bded32ef5b3df6c712948837f"} Nov 02 08:28:19.579524 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.579404 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" Nov 02 08:28:19.579524 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.579447 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz" event={"ID":"a1880d50-8522-44ed-bfff-a4567776155c","Type":"ContainerDied","Data":"5a18f81646f775470752bb92d6bef42535b6181ca70ca69a260705bb0c034098"} Nov 02 08:28:19.579524 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.579474 5899 scope.go:122] "RemoveContainer" containerID="5a820622339b516eaf601287ccc0f97268870d3bded32ef5b3df6c712948837f" Nov 02 08:28:19.595380 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.595357 5899 scope.go:122] "RemoveContainer" containerID="5a820622339b516eaf601287ccc0f97268870d3bded32ef5b3df6c712948837f" Nov 02 08:28:19.595893 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:28:19.595819 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a820622339b516eaf601287ccc0f97268870d3bded32ef5b3df6c712948837f\": container with ID starting with 5a820622339b516eaf601287ccc0f97268870d3bded32ef5b3df6c712948837f not found: ID does not exist" containerID="5a820622339b516eaf601287ccc0f97268870d3bded32ef5b3df6c712948837f" Nov 02 08:28:19.595893 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.595863 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a820622339b516eaf601287ccc0f97268870d3bded32ef5b3df6c712948837f"} err="failed to get container status \"5a820622339b516eaf601287ccc0f97268870d3bded32ef5b3df6c712948837f\": rpc error: code = NotFound desc = could not find container \"5a820622339b516eaf601287ccc0f97268870d3bded32ef5b3df6c712948837f\": container with ID starting with 5a820622339b516eaf601287ccc0f97268870d3bded32ef5b3df6c712948837f not found: ID does not exist" Nov 02 08:28:19.625407 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.625377 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:19.635907 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.635872 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz"] Nov 02 08:28:19.641362 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.641332 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb6010-5b6f9655b4-cgqvz"] Nov 02 08:28:19.820297 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.820221 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1880d50-8522-44ed-bfff-a4567776155c" path="/var/lib/kubelet/pods/a1880d50-8522-44ed-bfff-a4567776155c/volumes" Nov 02 08:28:19.946859 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:19.946830 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775"] Nov 02 08:28:19.948902 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:28:19.948874 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod053fdd58_57a4_42a5_9cd6_4891d05da186.slice/crio-6a636244950a4211bf30d5baee67cc350f0e163526ef27e060ca4612a1b7d5db WatchSource:0}: Error finding container 6a636244950a4211bf30d5baee67cc350f0e163526ef27e060ca4612a1b7d5db: Status 404 returned error can't find the container with id 6a636244950a4211bf30d5baee67cc350f0e163526ef27e060ca4612a1b7d5db Nov 02 08:28:20.601468 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:20.601430 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" event={"ID":"053fdd58-57a4-42a5-9cd6-4891d05da186","Type":"ContainerStarted","Data":"d3bee85d591775d0b4698ab0e2a1a78f48486ee09b416a5ecb01827365702dbe"} Nov 02 08:28:20.601979 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:20.601477 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" event={"ID":"053fdd58-57a4-42a5-9cd6-4891d05da186","Type":"ContainerStarted","Data":"6a636244950a4211bf30d5baee67cc350f0e163526ef27e060ca4612a1b7d5db"} Nov 02 08:28:20.626533 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:20.626504 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:20.655340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:20.655288 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" podStartSLOduration=1.655275428 podStartE2EDuration="1.655275428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:20.654256884 +0000 UTC m=+3800.952537719" watchObservedRunningTime="2026-11-02 08:28:20.655275428 +0000 UTC m=+3800.953556252" Nov 02 08:28:21.095383 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:21.095347 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:22.487739 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.487702 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:22.644537 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.644489 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:22.758285 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.758202 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9"] Nov 02 08:28:22.762601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.762579 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:22.784681 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.784659 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4b85d50c-serving-cert\"" Nov 02 08:28:22.797586 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.797552 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9"] Nov 02 08:28:22.875938 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.875906 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-maas-api-tls\") pod \"maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:22.876097 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.875969 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea559d19-13b8-438c-9988-52678b4dae8c-openshift-service-ca\") pod \"maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:22.876097 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.876085 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-metrics-certs\") pod \"maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:22.876241 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.876151 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggbsm\" (UniqueName: \"kubernetes.io/projected/ea559d19-13b8-438c-9988-52678b4dae8c-kube-api-access-ggbsm\") pod \"maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:22.890113 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.890078 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:22.977767 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.977726 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-metrics-certs\") pod \"maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:22.977928 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.977821 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggbsm\" (UniqueName: \"kubernetes.io/projected/ea559d19-13b8-438c-9988-52678b4dae8c-kube-api-access-ggbsm\") pod \"maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:22.977970 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.977925 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-maas-api-tls\") pod \"maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:22.978029 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.977971 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea559d19-13b8-438c-9988-52678b4dae8c-openshift-service-ca\") pod \"maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:22.978671 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.978647 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea559d19-13b8-438c-9988-52678b4dae8c-openshift-service-ca\") pod \"maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:22.980271 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.980248 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-metrics-certs\") pod \"maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:22.980549 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:22.980527 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-maas-api-tls\") pod \"maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:23.001928 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:23.001901 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggbsm\" (UniqueName: \"kubernetes.io/projected/ea559d19-13b8-438c-9988-52678b4dae8c-kube-api-access-ggbsm\") pod \"maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:23.029130 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:23.029065 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:23.077422 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:23.077382 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:23.158550 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:23.158491 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:23.309592 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:23.309485 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:23.394899 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:23.394868 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9"] Nov 02 08:28:23.396666 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:28:23.396639 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea559d19_13b8_438c_9988_52678b4dae8c.slice/crio-e1dbd590e65e38d1e1579989d41bbcb61fd03ab9ace5cd05a05d37708649bd30 WatchSource:0}: Error finding container e1dbd590e65e38d1e1579989d41bbcb61fd03ab9ace5cd05a05d37708649bd30: Status 404 returned error can't find the container with id e1dbd590e65e38d1e1579989d41bbcb61fd03ab9ace5cd05a05d37708649bd30 Nov 02 08:28:23.398063 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:23.398043 5899 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:28:23.622890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:23.622800 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" event={"ID":"ea559d19-13b8-438c-9988-52678b4dae8c","Type":"ContainerStarted","Data":"e1dbd590e65e38d1e1579989d41bbcb61fd03ab9ace5cd05a05d37708649bd30"} Nov 02 08:28:23.675785 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:23.675752 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:24.513464 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:24.513430 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:24.633944 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:24.633899 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" event={"ID":"ea559d19-13b8-438c-9988-52678b4dae8c","Type":"ContainerStarted","Data":"a79eecee447da2879660ddae27858cb8f0803cb8f81477953df5a39c2b9405c7"} Nov 02 08:28:24.634440 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:24.634106 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:24.656415 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:24.656381 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:24.679596 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:24.679542 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" podStartSLOduration=2.237360201 podStartE2EDuration="2.67952661s" totalImagesPullingTime="442.166409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:28:24.678151785 +0000 UTC m=+3804.976432613" watchObservedRunningTime="2026-11-02 08:28:24.67952661 +0000 UTC m=+3804.977807434" Nov 02 08:28:24.830402 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:24.830321 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:25.128302 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:25.128209 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:26.899591 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:26.899555 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:28.612910 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:28.612863 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:29.626224 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:29.626191 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:29.639974 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:29.639951 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:29.677784 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:29.677755 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:28:30.516994 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:30.516955 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:30.646061 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:30.646029 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:28:31.534390 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:31.534331 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:31.698915 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:31.698881 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:32.021938 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:32.021901 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:32.297586 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:32.297500 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:34.114659 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:34.114591 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:35.899389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:35.899347 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:37.078095 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:37.078054 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb"] Nov 02 08:28:37.242510 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:37.242470 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:37.384310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:37.384226 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:37.839322 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:37.839287 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:39.008185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:39.008105 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:39.600153 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:39.600112 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:41.377450 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:41.377420 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:42.078688 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:42.078637 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" podUID="654cd8f0-8663-470c-bcc2-26a96862186d" containerName="payload-processing" containerID="cri-o://704f5c3ab130ce383014d819e85521be9fc28adf0134cfe1823b5841371828e9" gracePeriod=25 Nov 02 08:28:42.725901 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:42.725863 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:42.906395 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:42.906365 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:43.147391 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:28:43.147355 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:28:43.265844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:43.265799 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:45.017941 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:45.017898 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:45.149990 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:45.149956 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:45.582302 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:45.582264 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:45.614514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:45.614468 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" podUID="654cd8f0-8663-470c-bcc2-26a96862186d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:28:46.913329 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:46.913290 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:48.796083 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:48.796046 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:50.754840 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:50.754802 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:52.614759 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:52.614718 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:54.385559 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:54.385516 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:55.614495 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:55.614455 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" podUID="654cd8f0-8663-470c-bcc2-26a96862186d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:28:55.765376 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:55.765332 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6"] Nov 02 08:28:55.765596 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:55.765567 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" podUID="6681e487-e123-40f2-82f7-9eb3eaac860b" containerName="maas-api" containerID="cri-o://bcb8e837f9547bcff109b5ad6fc0d280b5302dc35e2d4b659eeeb3931c74b26d" gracePeriod=30 Nov 02 08:28:55.829875 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:55.829824 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" podUID="6681e487-e123-40f2-82f7-9eb3eaac860b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.116:8443/health\": dial tcp 10.134.0.116:8443: connect: connection refused" Nov 02 08:28:55.859771 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:55.859729 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:56.044492 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.044456 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:28:56.125559 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.125513 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-metrics-certs\") pod \"6681e487-e123-40f2-82f7-9eb3eaac860b\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " Nov 02 08:28:56.125768 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.125608 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6681e487-e123-40f2-82f7-9eb3eaac860b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6681e487-e123-40f2-82f7-9eb3eaac860b-openshift-service-ca\") pod \"6681e487-e123-40f2-82f7-9eb3eaac860b\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " Nov 02 08:28:56.125869 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.125849 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6681e487-e123-40f2-82f7-9eb3eaac860b-kube-api-access-6nq5f\" (UniqueName: \"kubernetes.io/projected/6681e487-e123-40f2-82f7-9eb3eaac860b-kube-api-access-6nq5f\") pod \"6681e487-e123-40f2-82f7-9eb3eaac860b\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " Nov 02 08:28:56.125951 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.125915 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-maas-api-tls\") pod \"6681e487-e123-40f2-82f7-9eb3eaac860b\" (UID: \"6681e487-e123-40f2-82f7-9eb3eaac860b\") " Nov 02 08:28:56.126039 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.126006 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6681e487-e123-40f2-82f7-9eb3eaac860b-openshift-service-ca" pod "6681e487-e123-40f2-82f7-9eb3eaac860b" (UID: "6681e487-e123-40f2-82f7-9eb3eaac860b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:28:56.126536 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.126509 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6681e487-e123-40f2-82f7-9eb3eaac860b-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:28:56.133261 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.128304 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-metrics-certs" pod "6681e487-e123-40f2-82f7-9eb3eaac860b" (UID: "6681e487-e123-40f2-82f7-9eb3eaac860b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:28:56.133261 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.128600 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-maas-api-tls" pod "6681e487-e123-40f2-82f7-9eb3eaac860b" (UID: "6681e487-e123-40f2-82f7-9eb3eaac860b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:28:56.133261 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.130648 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6681e487-e123-40f2-82f7-9eb3eaac860b-kube-api-access-6nq5f" pod "6681e487-e123-40f2-82f7-9eb3eaac860b" (UID: "6681e487-e123-40f2-82f7-9eb3eaac860b"). InnerVolumeSpecName "kube-api-access-6nq5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:28:56.228284 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.228245 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nq5f\" (UniqueName: \"kubernetes.io/projected/6681e487-e123-40f2-82f7-9eb3eaac860b-kube-api-access-6nq5f\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:28:56.228284 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.228282 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:28:56.228284 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.228291 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6681e487-e123-40f2-82f7-9eb3eaac860b-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:28:56.626430 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.626395 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:28:56.855262 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.855221 5899 generic.go:356] "Generic (PLEG): container finished" podID="6681e487-e123-40f2-82f7-9eb3eaac860b" containerID="bcb8e837f9547bcff109b5ad6fc0d280b5302dc35e2d4b659eeeb3931c74b26d" exitCode=0 Nov 02 08:28:56.855262 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.855267 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" Nov 02 08:28:56.855470 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.855301 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" event={"ID":"6681e487-e123-40f2-82f7-9eb3eaac860b","Type":"ContainerDied","Data":"bcb8e837f9547bcff109b5ad6fc0d280b5302dc35e2d4b659eeeb3931c74b26d"} Nov 02 08:28:56.855470 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.855355 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6" event={"ID":"6681e487-e123-40f2-82f7-9eb3eaac860b","Type":"ContainerDied","Data":"014f0ee491857c4eb7e631543d5a1c6bcb5fdee6eb34c5eacfeefc11d10561d6"} Nov 02 08:28:56.855470 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.855376 5899 scope.go:122] "RemoveContainer" containerID="bcb8e837f9547bcff109b5ad6fc0d280b5302dc35e2d4b659eeeb3931c74b26d" Nov 02 08:28:56.866694 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.866672 5899 scope.go:122] "RemoveContainer" containerID="bcb8e837f9547bcff109b5ad6fc0d280b5302dc35e2d4b659eeeb3931c74b26d" Nov 02 08:28:56.867016 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:28:56.866995 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb8e837f9547bcff109b5ad6fc0d280b5302dc35e2d4b659eeeb3931c74b26d\": container with ID starting with bcb8e837f9547bcff109b5ad6fc0d280b5302dc35e2d4b659eeeb3931c74b26d not found: ID does not exist" containerID="bcb8e837f9547bcff109b5ad6fc0d280b5302dc35e2d4b659eeeb3931c74b26d" Nov 02 08:28:56.867065 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.867027 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb8e837f9547bcff109b5ad6fc0d280b5302dc35e2d4b659eeeb3931c74b26d"} err="failed to get container status \"bcb8e837f9547bcff109b5ad6fc0d280b5302dc35e2d4b659eeeb3931c74b26d\": rpc error: code = NotFound desc = could not find container \"bcb8e837f9547bcff109b5ad6fc0d280b5302dc35e2d4b659eeeb3931c74b26d\": container with ID starting with bcb8e837f9547bcff109b5ad6fc0d280b5302dc35e2d4b659eeeb3931c74b26d not found: ID does not exist" Nov 02 08:28:56.912451 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.912377 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6"] Nov 02 08:28:56.925811 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:56.925775 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-736f7c-5df47fcb9c-q8zr6"] Nov 02 08:28:57.820418 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:57.820384 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6681e487-e123-40f2-82f7-9eb3eaac860b" path="/var/lib/kubelet/pods/6681e487-e123-40f2-82f7-9eb3eaac860b/volumes" Nov 02 08:28:58.519649 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:28:58.519607 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:00.490991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:00.490956 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:01.810396 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:01.810362 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:01.956262 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:01.956227 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:02.106497 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.106462 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:02.357041 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.356970 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" Nov 02 08:29:02.405329 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.405299 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654cd8f0-8663-470c-bcc2-26a96862186d-kube-api-access-g82gf\" (UniqueName: \"kubernetes.io/projected/654cd8f0-8663-470c-bcc2-26a96862186d-kube-api-access-g82gf\") pod \"654cd8f0-8663-470c-bcc2-26a96862186d\" (UID: \"654cd8f0-8663-470c-bcc2-26a96862186d\") " Nov 02 08:29:02.405526 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.405451 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/654cd8f0-8663-470c-bcc2-26a96862186d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654cd8f0-8663-470c-bcc2-26a96862186d-plugins-config-volume\") pod \"654cd8f0-8663-470c-bcc2-26a96862186d\" (UID: \"654cd8f0-8663-470c-bcc2-26a96862186d\") " Nov 02 08:29:02.405933 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.405905 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654cd8f0-8663-470c-bcc2-26a96862186d-plugins-config-volume" pod "654cd8f0-8663-470c-bcc2-26a96862186d" (UID: "654cd8f0-8663-470c-bcc2-26a96862186d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:29:02.407822 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.407795 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654cd8f0-8663-470c-bcc2-26a96862186d-kube-api-access-g82gf" pod "654cd8f0-8663-470c-bcc2-26a96862186d" (UID: "654cd8f0-8663-470c-bcc2-26a96862186d"). InnerVolumeSpecName "kube-api-access-g82gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:29:02.507332 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.507294 5899 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654cd8f0-8663-470c-bcc2-26a96862186d-plugins-config-volume\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:29:02.507332 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.507323 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g82gf\" (UniqueName: \"kubernetes.io/projected/654cd8f0-8663-470c-bcc2-26a96862186d-kube-api-access-g82gf\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:29:02.676304 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.676203 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:02.898842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.898806 5899 generic.go:356] "Generic (PLEG): container finished" podID="654cd8f0-8663-470c-bcc2-26a96862186d" containerID="704f5c3ab130ce383014d819e85521be9fc28adf0134cfe1823b5841371828e9" exitCode=0 Nov 02 08:29:02.899291 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.898871 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" Nov 02 08:29:02.899291 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.898878 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" event={"ID":"654cd8f0-8663-470c-bcc2-26a96862186d","Type":"ContainerDied","Data":"704f5c3ab130ce383014d819e85521be9fc28adf0134cfe1823b5841371828e9"} Nov 02 08:29:02.899291 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.899000 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb" event={"ID":"654cd8f0-8663-470c-bcc2-26a96862186d","Type":"ContainerDied","Data":"cc0c659d07a9e42448559a9e09e52adf68a78e9e9ef47fcd36c563c9ba2913ab"} Nov 02 08:29:02.899291 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.899023 5899 scope.go:122] "RemoveContainer" containerID="704f5c3ab130ce383014d819e85521be9fc28adf0134cfe1823b5841371828e9" Nov 02 08:29:02.912150 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.912122 5899 scope.go:122] "RemoveContainer" containerID="704f5c3ab130ce383014d819e85521be9fc28adf0134cfe1823b5841371828e9" Nov 02 08:29:02.912441 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:29:02.912420 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704f5c3ab130ce383014d819e85521be9fc28adf0134cfe1823b5841371828e9\": container with ID starting with 704f5c3ab130ce383014d819e85521be9fc28adf0134cfe1823b5841371828e9 not found: ID does not exist" containerID="704f5c3ab130ce383014d819e85521be9fc28adf0134cfe1823b5841371828e9" Nov 02 08:29:02.912511 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.912448 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704f5c3ab130ce383014d819e85521be9fc28adf0134cfe1823b5841371828e9"} err="failed to get container status \"704f5c3ab130ce383014d819e85521be9fc28adf0134cfe1823b5841371828e9\": rpc error: code = NotFound desc = could not find container \"704f5c3ab130ce383014d819e85521be9fc28adf0134cfe1823b5841371828e9\": container with ID starting with 704f5c3ab130ce383014d819e85521be9fc28adf0134cfe1823b5841371828e9 not found: ID does not exist" Nov 02 08:29:02.965974 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.965894 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb"] Nov 02 08:29:02.979008 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:02.978964 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-736f7c-b8cfc9767-k49mb"] Nov 02 08:29:03.127424 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:03.127382 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:03.820339 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:03.820307 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="654cd8f0-8663-470c-bcc2-26a96862186d" path="/var/lib/kubelet/pods/654cd8f0-8663-470c-bcc2-26a96862186d/volumes" Nov 02 08:29:03.822173 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:03.822138 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:04.250469 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:04.250437 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:06.103603 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:06.103561 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:07.852257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:07.852224 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:08.850286 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:08.850253 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:09.843905 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:09.843871 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:10.641256 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:10.641215 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:10.792303 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:10.792270 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:10.932412 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:10.932316 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:11.743423 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:11.743388 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:13.141672 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:29:13.141640 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:29:13.494037 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:13.494000 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:14.567568 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:14.567526 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:14.704407 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:14.704373 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:15.229056 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:15.229021 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:15.359825 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:15.359790 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:16.078560 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:16.078521 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:16.253557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:16.253521 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:16.674502 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:16.674459 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:17.052330 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:17.052287 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:17.205769 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:17.205735 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:18.813420 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:18.813388 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:20.452890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:20.452861 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:20.594879 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:20.594841 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:20.773649 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:20.773616 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:22.128068 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:22.128033 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:23.253737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:23.253699 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:23.903630 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:23.903599 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:25.611684 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:25.611647 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:26.611520 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:26.611480 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:26.816657 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:26.816598 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:26.991743 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:26.991711 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:27.589134 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:27.589099 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:29.342145 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:29.342113 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:31.076800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:31.076742 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:32.636520 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:32.636483 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:32.849652 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:32.849615 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:34.505940 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:34.505903 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:36.295626 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:36.295587 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:37.414604 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:37.414567 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:38.025706 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:38.025677 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:39.701494 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:39.701456 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:41.412724 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:41.412646 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:43.143452 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:29:43.143415 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:29:43.209320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:43.209282 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:43.460342 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:43.460257 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:44.902652 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:44.902615 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:46.591961 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:46.591920 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:48.290997 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:48.290958 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:50.005627 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:50.005592 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:51.507683 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:51.507643 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4"] Nov 02 08:29:51.508241 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:51.507889 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" podUID="7b327faf-1f96-4df8-b747-f2b810fa3ae4" containerName="maas-api" containerID="cri-o://c6bfb3e11596ff1a87b039dfb6798feddfeef71fff9e937a4e8a992d7028adfc" gracePeriod=30 Nov 02 08:29:51.572956 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:51.572881 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:51.807039 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:51.807010 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:29:51.936269 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:51.936231 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-metrics-certs\") pod \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " Nov 02 08:29:51.936446 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:51.936325 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b327faf-1f96-4df8-b747-f2b810fa3ae4-kube-api-access-9t6qh\" (UniqueName: \"kubernetes.io/projected/7b327faf-1f96-4df8-b747-f2b810fa3ae4-kube-api-access-9t6qh\") pod \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " Nov 02 08:29:51.936446 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:51.936390 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b327faf-1f96-4df8-b747-f2b810fa3ae4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b327faf-1f96-4df8-b747-f2b810fa3ae4-openshift-service-ca\") pod \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " Nov 02 08:29:51.936446 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:51.936439 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-maas-api-tls\") pod \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\" (UID: \"7b327faf-1f96-4df8-b747-f2b810fa3ae4\") " Nov 02 08:29:51.936835 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:51.936799 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b327faf-1f96-4df8-b747-f2b810fa3ae4-openshift-service-ca" pod "7b327faf-1f96-4df8-b747-f2b810fa3ae4" (UID: "7b327faf-1f96-4df8-b747-f2b810fa3ae4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:29:51.937086 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:51.937066 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b327faf-1f96-4df8-b747-f2b810fa3ae4-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:29:51.938534 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:51.938505 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-maas-api-tls" pod "7b327faf-1f96-4df8-b747-f2b810fa3ae4" (UID: "7b327faf-1f96-4df8-b747-f2b810fa3ae4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:29:51.938641 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:51.938551 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-metrics-certs" pod "7b327faf-1f96-4df8-b747-f2b810fa3ae4" (UID: "7b327faf-1f96-4df8-b747-f2b810fa3ae4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:29:51.938686 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:51.938673 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b327faf-1f96-4df8-b747-f2b810fa3ae4-kube-api-access-9t6qh" pod "7b327faf-1f96-4df8-b747-f2b810fa3ae4" (UID: "7b327faf-1f96-4df8-b747-f2b810fa3ae4"). InnerVolumeSpecName "kube-api-access-9t6qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:29:52.038557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:52.038465 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:29:52.038557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:52.038501 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t6qh\" (UniqueName: \"kubernetes.io/projected/7b327faf-1f96-4df8-b747-f2b810fa3ae4-kube-api-access-9t6qh\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:29:52.038557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:52.038515 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b327faf-1f96-4df8-b747-f2b810fa3ae4-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:29:52.210611 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:52.210576 5899 generic.go:356] "Generic (PLEG): container finished" podID="7b327faf-1f96-4df8-b747-f2b810fa3ae4" containerID="c6bfb3e11596ff1a87b039dfb6798feddfeef71fff9e937a4e8a992d7028adfc" exitCode=0 Nov 02 08:29:52.210821 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:52.210627 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" event={"ID":"7b327faf-1f96-4df8-b747-f2b810fa3ae4","Type":"ContainerDied","Data":"c6bfb3e11596ff1a87b039dfb6798feddfeef71fff9e937a4e8a992d7028adfc"} Nov 02 08:29:52.210821 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:52.210685 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" event={"ID":"7b327faf-1f96-4df8-b747-f2b810fa3ae4","Type":"ContainerDied","Data":"2ba86d730fb2037106a300b8cdc3880ac79fe822a772ae87fd6cc0b3714bf2ee"} Nov 02 08:29:52.210821 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:52.210637 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4" Nov 02 08:29:52.210821 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:52.210728 5899 scope.go:122] "RemoveContainer" containerID="c6bfb3e11596ff1a87b039dfb6798feddfeef71fff9e937a4e8a992d7028adfc" Nov 02 08:29:52.222457 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:52.222194 5899 scope.go:122] "RemoveContainer" containerID="c6bfb3e11596ff1a87b039dfb6798feddfeef71fff9e937a4e8a992d7028adfc" Nov 02 08:29:52.222758 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:29:52.222567 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6bfb3e11596ff1a87b039dfb6798feddfeef71fff9e937a4e8a992d7028adfc\": container with ID starting with c6bfb3e11596ff1a87b039dfb6798feddfeef71fff9e937a4e8a992d7028adfc not found: ID does not exist" containerID="c6bfb3e11596ff1a87b039dfb6798feddfeef71fff9e937a4e8a992d7028adfc" Nov 02 08:29:52.222758 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:52.222606 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6bfb3e11596ff1a87b039dfb6798feddfeef71fff9e937a4e8a992d7028adfc"} err="failed to get container status \"c6bfb3e11596ff1a87b039dfb6798feddfeef71fff9e937a4e8a992d7028adfc\": rpc error: code = NotFound desc = could not find container \"c6bfb3e11596ff1a87b039dfb6798feddfeef71fff9e937a4e8a992d7028adfc\": container with ID starting with c6bfb3e11596ff1a87b039dfb6798feddfeef71fff9e937a4e8a992d7028adfc not found: ID does not exist" Nov 02 08:29:52.265425 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:52.265393 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4"] Nov 02 08:29:52.268093 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:52.268064 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dde84-57cdb9995f-fhnx4"] Nov 02 08:29:52.302920 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:52.302833 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:53.820454 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:53.820423 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b327faf-1f96-4df8-b747-f2b810fa3ae4" path="/var/lib/kubelet/pods/7b327faf-1f96-4df8-b747-f2b810fa3ae4/volumes" Nov 02 08:29:53.975291 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:53.975257 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:55.717220 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:55.717176 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:57.428649 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:57.428612 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:29:59.289778 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:29:59.289740 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:00.766493 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:00.766458 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:00.911877 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:00.911841 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:01.212563 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:01.212477 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:01.524751 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:01.524690 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:02.035373 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:02.035338 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:02.779472 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:02.779415 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:04.343221 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:04.343192 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:06.328514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:06.328474 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:07.700779 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:07.700741 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf"] Nov 02 08:30:07.701343 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:07.700978 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" podUID="b057d2fb-2171-4165-8185-070d04931f5f" containerName="maas-api" containerID="cri-o://51cb6aa8c9ec9fb09fa2beb31547a38a86a96afcd44626e3402dbb87100c910e" gracePeriod=30 Nov 02 08:30:07.817533 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:07.817490 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:07.983777 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:07.983748 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:30:08.111502 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.111461 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-metrics-certs\") pod \"b057d2fb-2171-4165-8185-070d04931f5f\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " Nov 02 08:30:08.111712 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.111610 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b057d2fb-2171-4165-8185-070d04931f5f-kube-api-access-vwmwx\" (UniqueName: \"kubernetes.io/projected/b057d2fb-2171-4165-8185-070d04931f5f-kube-api-access-vwmwx\") pod \"b057d2fb-2171-4165-8185-070d04931f5f\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " Nov 02 08:30:08.111712 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.111680 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-maas-api-tls\") pod \"b057d2fb-2171-4165-8185-070d04931f5f\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " Nov 02 08:30:08.111712 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.111700 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b057d2fb-2171-4165-8185-070d04931f5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b057d2fb-2171-4165-8185-070d04931f5f-openshift-service-ca\") pod \"b057d2fb-2171-4165-8185-070d04931f5f\" (UID: \"b057d2fb-2171-4165-8185-070d04931f5f\") " Nov 02 08:30:08.112207 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.112180 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b057d2fb-2171-4165-8185-070d04931f5f-openshift-service-ca" pod "b057d2fb-2171-4165-8185-070d04931f5f" (UID: "b057d2fb-2171-4165-8185-070d04931f5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:30:08.114002 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.113803 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-maas-api-tls" pod "b057d2fb-2171-4165-8185-070d04931f5f" (UID: "b057d2fb-2171-4165-8185-070d04931f5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:30:08.114273 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.114246 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-metrics-certs" pod "b057d2fb-2171-4165-8185-070d04931f5f" (UID: "b057d2fb-2171-4165-8185-070d04931f5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:30:08.114356 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.114337 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b057d2fb-2171-4165-8185-070d04931f5f-kube-api-access-vwmwx" pod "b057d2fb-2171-4165-8185-070d04931f5f" (UID: "b057d2fb-2171-4165-8185-070d04931f5f"). InnerVolumeSpecName "kube-api-access-vwmwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:08.213995 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.213888 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:30:08.213995 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.213932 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b057d2fb-2171-4165-8185-070d04931f5f-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:30:08.213995 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.213947 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b057d2fb-2171-4165-8185-070d04931f5f-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:30:08.213995 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.213961 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwmwx\" (UniqueName: \"kubernetes.io/projected/b057d2fb-2171-4165-8185-070d04931f5f-kube-api-access-vwmwx\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:30:08.335050 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.335015 5899 generic.go:356] "Generic (PLEG): container finished" podID="b057d2fb-2171-4165-8185-070d04931f5f" containerID="51cb6aa8c9ec9fb09fa2beb31547a38a86a96afcd44626e3402dbb87100c910e" exitCode=0 Nov 02 08:30:08.335251 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.335067 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" Nov 02 08:30:08.335251 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.335085 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" event={"ID":"b057d2fb-2171-4165-8185-070d04931f5f","Type":"ContainerDied","Data":"51cb6aa8c9ec9fb09fa2beb31547a38a86a96afcd44626e3402dbb87100c910e"} Nov 02 08:30:08.335251 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.335128 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf" event={"ID":"b057d2fb-2171-4165-8185-070d04931f5f","Type":"ContainerDied","Data":"b6dcafcbb80fd94e0a2e4f43d23798592dc5f243f57d5216533977a98c6b4546"} Nov 02 08:30:08.335251 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.335150 5899 scope.go:122] "RemoveContainer" containerID="51cb6aa8c9ec9fb09fa2beb31547a38a86a96afcd44626e3402dbb87100c910e" Nov 02 08:30:08.346650 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.346622 5899 scope.go:122] "RemoveContainer" containerID="51cb6aa8c9ec9fb09fa2beb31547a38a86a96afcd44626e3402dbb87100c910e" Nov 02 08:30:08.346955 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:30:08.346933 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51cb6aa8c9ec9fb09fa2beb31547a38a86a96afcd44626e3402dbb87100c910e\": container with ID starting with 51cb6aa8c9ec9fb09fa2beb31547a38a86a96afcd44626e3402dbb87100c910e not found: ID does not exist" containerID="51cb6aa8c9ec9fb09fa2beb31547a38a86a96afcd44626e3402dbb87100c910e" Nov 02 08:30:08.347007 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.346963 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cb6aa8c9ec9fb09fa2beb31547a38a86a96afcd44626e3402dbb87100c910e"} err="failed to get container status \"51cb6aa8c9ec9fb09fa2beb31547a38a86a96afcd44626e3402dbb87100c910e\": rpc error: code = NotFound desc = could not find container \"51cb6aa8c9ec9fb09fa2beb31547a38a86a96afcd44626e3402dbb87100c910e\": container with ID starting with 51cb6aa8c9ec9fb09fa2beb31547a38a86a96afcd44626e3402dbb87100c910e not found: ID does not exist" Nov 02 08:30:08.386355 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.386319 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf"] Nov 02 08:30:08.397579 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.397551 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f2e06-bdc47f6d-x7zrf"] Nov 02 08:30:08.606254 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.606217 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:08.970429 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:08.970325 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:09.117681 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:09.117643 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:09.260127 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:09.260094 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:09.831184 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:09.831130 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b057d2fb-2171-4165-8185-070d04931f5f" path="/var/lib/kubelet/pods/b057d2fb-2171-4165-8185-070d04931f5f/volumes" Nov 02 08:30:10.138663 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:10.138564 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:10.290595 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:10.290561 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:10.438773 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:10.438661 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:10.569458 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:10.569400 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:10.860331 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:10.860296 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:11.219397 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:11.219287 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:11.960500 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:11.960461 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:12.470056 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:12.470016 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:13.141495 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:30:13.141459 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:30:14.148146 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:14.148107 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:15.801613 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:15.801576 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:16.333719 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:16.333684 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:17.658791 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:17.658752 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:19.245601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:19.245563 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:19.541771 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:19.541685 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:19.689849 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:19.689807 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:19.844344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:19.844234 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:21.511395 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:21.511362 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:23.312107 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:23.312061 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:25.190480 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:25.190443 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:25.884421 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:25.884383 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:26.937794 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:26.937758 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:28.714663 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:28.714611 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:30.551202 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:30.551148 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:32.322983 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:32.322947 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:33.983397 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:33.983360 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:35.736147 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:35.736105 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:37.396957 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:37.396906 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:39.077432 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:39.077390 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:41.016671 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:41.016637 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:42.402794 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:42.402700 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:43.051092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:43.051052 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:43.143679 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:30:43.143638 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:30:44.317387 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:44.317348 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:44.454056 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:44.454018 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:44.709304 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:44.709210 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:46.425344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:46.425302 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:47.498987 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:47.498952 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:48.218599 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:48.218561 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:49.911633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:49.911595 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:51.304934 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.304896 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl"] Nov 02 08:30:51.306067 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.306008 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" podUID="760654b4-d894-489c-bf1b-3dd442fbe8da" containerName="maas-api" containerID="cri-o://568a4fde7b923ec39355beefba74136998ed30643f3b1d5d61cca5562a1d2481" gracePeriod=30 Nov 02 08:30:51.402049 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.401977 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:51.607278 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.607244 5899 generic.go:356] "Generic (PLEG): container finished" podID="760654b4-d894-489c-bf1b-3dd442fbe8da" containerID="568a4fde7b923ec39355beefba74136998ed30643f3b1d5d61cca5562a1d2481" exitCode=0 Nov 02 08:30:51.607466 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.607326 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" event={"ID":"760654b4-d894-489c-bf1b-3dd442fbe8da","Type":"ContainerDied","Data":"568a4fde7b923ec39355beefba74136998ed30643f3b1d5d61cca5562a1d2481"} Nov 02 08:30:51.607466 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.607381 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" event={"ID":"760654b4-d894-489c-bf1b-3dd442fbe8da","Type":"ContainerDied","Data":"503758b263877c966d62893234ec0e38d104a22a11578815e2fa5bd36198fbc1"} Nov 02 08:30:51.607466 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.607403 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="503758b263877c966d62893234ec0e38d104a22a11578815e2fa5bd36198fbc1" Nov 02 08:30:51.615517 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.615480 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:30:51.663983 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.663942 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-maas-api-tls\") pod \"760654b4-d894-489c-bf1b-3dd442fbe8da\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " Nov 02 08:30:51.664186 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.664044 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-metrics-certs\") pod \"760654b4-d894-489c-bf1b-3dd442fbe8da\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " Nov 02 08:30:51.664186 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.664110 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/760654b4-d894-489c-bf1b-3dd442fbe8da-kube-api-access-zc598\" (UniqueName: \"kubernetes.io/projected/760654b4-d894-489c-bf1b-3dd442fbe8da-kube-api-access-zc598\") pod \"760654b4-d894-489c-bf1b-3dd442fbe8da\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " Nov 02 08:30:51.664334 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.664187 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/760654b4-d894-489c-bf1b-3dd442fbe8da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760654b4-d894-489c-bf1b-3dd442fbe8da-openshift-service-ca\") pod \"760654b4-d894-489c-bf1b-3dd442fbe8da\" (UID: \"760654b4-d894-489c-bf1b-3dd442fbe8da\") " Nov 02 08:30:51.665371 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.665320 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760654b4-d894-489c-bf1b-3dd442fbe8da-openshift-service-ca" pod "760654b4-d894-489c-bf1b-3dd442fbe8da" (UID: "760654b4-d894-489c-bf1b-3dd442fbe8da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:30:51.672089 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.672032 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760654b4-d894-489c-bf1b-3dd442fbe8da-kube-api-access-zc598" pod "760654b4-d894-489c-bf1b-3dd442fbe8da" (UID: "760654b4-d894-489c-bf1b-3dd442fbe8da"). InnerVolumeSpecName "kube-api-access-zc598". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:30:51.672401 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.672379 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-metrics-certs" pod "760654b4-d894-489c-bf1b-3dd442fbe8da" (UID: "760654b4-d894-489c-bf1b-3dd442fbe8da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:30:51.674631 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.674605 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-maas-api-tls" pod "760654b4-d894-489c-bf1b-3dd442fbe8da" (UID: "760654b4-d894-489c-bf1b-3dd442fbe8da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:30:51.765555 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.765524 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc598\" (UniqueName: \"kubernetes.io/projected/760654b4-d894-489c-bf1b-3dd442fbe8da-kube-api-access-zc598\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:30:51.765555 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.765552 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/760654b4-d894-489c-bf1b-3dd442fbe8da-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:30:51.765555 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.765563 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:30:51.765798 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.765572 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/760654b4-d894-489c-bf1b-3dd442fbe8da-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:30:51.893406 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:51.893369 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:52.097408 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:52.097321 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:52.613846 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:52.613815 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl" Nov 02 08:30:52.634918 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:52.634890 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:52.666244 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:52.666214 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl"] Nov 02 08:30:52.677792 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:52.677758 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2365fa-54b7497765-x95tl"] Nov 02 08:30:53.107909 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:53.107878 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:53.258425 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:53.258385 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:53.820774 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:53.820738 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="760654b4-d894-489c-bf1b-3dd442fbe8da" path="/var/lib/kubelet/pods/760654b4-d894-489c-bf1b-3dd442fbe8da/volumes" Nov 02 08:30:53.900110 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:53.900069 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:54.391092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:54.391052 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:55.693650 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:55.693614 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:56.269658 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:56.269624 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:57.904474 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:57.904439 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:30:59.708235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:30:59.708201 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:00.563800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:00.563771 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:00.720846 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:00.720814 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:01.007981 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:01.007946 5899 scope.go:122] "RemoveContainer" containerID="568a4fde7b923ec39355beefba74136998ed30643f3b1d5d61cca5562a1d2481" Nov 02 08:31:01.682573 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:01.682536 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:01.812796 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:01.812751 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:02.226230 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:02.226193 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:02.395617 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:02.395574 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:02.987056 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:02.987013 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:03.126237 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:03.126186 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:03.389001 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:03.388905 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:03.628775 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:03.628736 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:04.155056 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:04.155018 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:05.349156 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:05.349119 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:07.102092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:07.102059 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:07.319852 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:07.319808 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:08.415652 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:08.415616 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:08.661264 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:08.661227 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9"] Nov 02 08:31:08.661914 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:08.661881 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" podUID="ea559d19-13b8-438c-9988-52678b4dae8c" containerName="maas-api" containerID="cri-o://a79eecee447da2879660ddae27858cb8f0803cb8f81477953df5a39c2b9405c7" gracePeriod=30 Nov 02 08:31:08.756304 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:08.756267 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:08.955465 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:08.955022 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:31:08.957897 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:08.957866 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:09.010644 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.010609 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-metrics-certs\") pod \"ea559d19-13b8-438c-9988-52678b4dae8c\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " Nov 02 08:31:09.010850 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.010670 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea559d19-13b8-438c-9988-52678b4dae8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea559d19-13b8-438c-9988-52678b4dae8c-openshift-service-ca\") pod \"ea559d19-13b8-438c-9988-52678b4dae8c\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " Nov 02 08:31:09.010850 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.010704 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-maas-api-tls\") pod \"ea559d19-13b8-438c-9988-52678b4dae8c\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " Nov 02 08:31:09.010850 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.010773 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea559d19-13b8-438c-9988-52678b4dae8c-kube-api-access-ggbsm\" (UniqueName: \"kubernetes.io/projected/ea559d19-13b8-438c-9988-52678b4dae8c-kube-api-access-ggbsm\") pod \"ea559d19-13b8-438c-9988-52678b4dae8c\" (UID: \"ea559d19-13b8-438c-9988-52678b4dae8c\") " Nov 02 08:31:09.011537 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.011506 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea559d19-13b8-438c-9988-52678b4dae8c-openshift-service-ca" pod "ea559d19-13b8-438c-9988-52678b4dae8c" (UID: "ea559d19-13b8-438c-9988-52678b4dae8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:31:09.014270 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.014241 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-metrics-certs" pod "ea559d19-13b8-438c-9988-52678b4dae8c" (UID: "ea559d19-13b8-438c-9988-52678b4dae8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:31:09.014808 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.014784 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea559d19-13b8-438c-9988-52678b4dae8c-kube-api-access-ggbsm" pod "ea559d19-13b8-438c-9988-52678b4dae8c" (UID: "ea559d19-13b8-438c-9988-52678b4dae8c"). InnerVolumeSpecName "kube-api-access-ggbsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:31:09.017190 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.016207 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-maas-api-tls" pod "ea559d19-13b8-438c-9988-52678b4dae8c" (UID: "ea559d19-13b8-438c-9988-52678b4dae8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:31:09.107499 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.107461 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:09.112979 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.112952 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:09.113083 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.112984 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea559d19-13b8-438c-9988-52678b4dae8c-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:09.113083 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.112993 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea559d19-13b8-438c-9988-52678b4dae8c-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:09.113083 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.113002 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggbsm\" (UniqueName: \"kubernetes.io/projected/ea559d19-13b8-438c-9988-52678b4dae8c-kube-api-access-ggbsm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:09.249725 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.249684 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:09.416516 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.416453 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:09.589488 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.589391 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:09.730106 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.730069 5899 generic.go:356] "Generic (PLEG): container finished" podID="ea559d19-13b8-438c-9988-52678b4dae8c" containerID="a79eecee447da2879660ddae27858cb8f0803cb8f81477953df5a39c2b9405c7" exitCode=0 Nov 02 08:31:09.730322 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.730117 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" Nov 02 08:31:09.730322 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.730173 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" event={"ID":"ea559d19-13b8-438c-9988-52678b4dae8c","Type":"ContainerDied","Data":"a79eecee447da2879660ddae27858cb8f0803cb8f81477953df5a39c2b9405c7"} Nov 02 08:31:09.730322 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.730232 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9" event={"ID":"ea559d19-13b8-438c-9988-52678b4dae8c","Type":"ContainerDied","Data":"e1dbd590e65e38d1e1579989d41bbcb61fd03ab9ace5cd05a05d37708649bd30"} Nov 02 08:31:09.730322 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.730264 5899 scope.go:122] "RemoveContainer" containerID="a79eecee447da2879660ddae27858cb8f0803cb8f81477953df5a39c2b9405c7" Nov 02 08:31:09.746777 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.746748 5899 scope.go:122] "RemoveContainer" containerID="a79eecee447da2879660ddae27858cb8f0803cb8f81477953df5a39c2b9405c7" Nov 02 08:31:09.747153 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:31:09.747122 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79eecee447da2879660ddae27858cb8f0803cb8f81477953df5a39c2b9405c7\": container with ID starting with a79eecee447da2879660ddae27858cb8f0803cb8f81477953df5a39c2b9405c7 not found: ID does not exist" containerID="a79eecee447da2879660ddae27858cb8f0803cb8f81477953df5a39c2b9405c7" Nov 02 08:31:09.747336 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.747194 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79eecee447da2879660ddae27858cb8f0803cb8f81477953df5a39c2b9405c7"} err="failed to get container status \"a79eecee447da2879660ddae27858cb8f0803cb8f81477953df5a39c2b9405c7\": rpc error: code = NotFound desc = could not find container \"a79eecee447da2879660ddae27858cb8f0803cb8f81477953df5a39c2b9405c7\": container with ID starting with a79eecee447da2879660ddae27858cb8f0803cb8f81477953df5a39c2b9405c7 not found: ID does not exist" Nov 02 08:31:09.779990 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.779956 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9"] Nov 02 08:31:09.791292 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.791261 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4b85d50c-65b4cd9464-lfsc9"] Nov 02 08:31:09.822078 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:09.822042 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea559d19-13b8-438c-9988-52678b4dae8c" path="/var/lib/kubelet/pods/ea559d19-13b8-438c-9988-52678b4dae8c/volumes" Nov 02 08:31:10.403956 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:10.403915 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:10.562535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:10.562490 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:11.352105 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:11.352069 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:11.522024 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:11.521989 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:11.698230 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:11.698139 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:11.846405 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:11.846368 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:12.870419 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:12.870382 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:13.082661 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.082628 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:13.143195 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:31:13.143100 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:31:13.594495 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.594456 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr"] Nov 02 08:31:13.595071 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595056 5899 cpu_manager.go:414] "containerMap: removing container" podUID="ea559d19-13b8-438c-9988-52678b4dae8c" containerName="maas-api" Nov 02 08:31:13.595119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595073 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea559d19-13b8-438c-9988-52678b4dae8c" containerName="maas-api" Nov 02 08:31:13.595119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595087 5899 cpu_manager.go:414] "containerMap: removing container" podUID="760654b4-d894-489c-bf1b-3dd442fbe8da" containerName="maas-api" Nov 02 08:31:13.595119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595096 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="760654b4-d894-489c-bf1b-3dd442fbe8da" containerName="maas-api" Nov 02 08:31:13.595119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595106 5899 cpu_manager.go:414] "containerMap: removing container" podUID="7b327faf-1f96-4df8-b747-f2b810fa3ae4" containerName="maas-api" Nov 02 08:31:13.595119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595113 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b327faf-1f96-4df8-b747-f2b810fa3ae4" containerName="maas-api" Nov 02 08:31:13.595119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595119 5899 cpu_manager.go:414] "containerMap: removing container" podUID="b057d2fb-2171-4165-8185-070d04931f5f" containerName="maas-api" Nov 02 08:31:13.595340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595124 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b057d2fb-2171-4165-8185-070d04931f5f" containerName="maas-api" Nov 02 08:31:13.595340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595140 5899 cpu_manager.go:414] "containerMap: removing container" podUID="654cd8f0-8663-470c-bcc2-26a96862186d" containerName="payload-processing" Nov 02 08:31:13.595340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595145 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654cd8f0-8663-470c-bcc2-26a96862186d" containerName="payload-processing" Nov 02 08:31:13.595340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595198 5899 cpu_manager.go:414] "containerMap: removing container" podUID="6681e487-e123-40f2-82f7-9eb3eaac860b" containerName="maas-api" Nov 02 08:31:13.595340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595204 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6681e487-e123-40f2-82f7-9eb3eaac860b" containerName="maas-api" Nov 02 08:31:13.595340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595290 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="654cd8f0-8663-470c-bcc2-26a96862186d" containerName="payload-processing" Nov 02 08:31:13.595340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595305 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="6681e487-e123-40f2-82f7-9eb3eaac860b" containerName="maas-api" Nov 02 08:31:13.595340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595312 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea559d19-13b8-438c-9988-52678b4dae8c" containerName="maas-api" Nov 02 08:31:13.595340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595319 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="760654b4-d894-489c-bf1b-3dd442fbe8da" containerName="maas-api" Nov 02 08:31:13.595340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595327 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b327faf-1f96-4df8-b747-f2b810fa3ae4" containerName="maas-api" Nov 02 08:31:13.595340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.595337 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="b057d2fb-2171-4165-8185-070d04931f5f" containerName="maas-api" Nov 02 08:31:13.600067 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.600044 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.616101 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.616073 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-12df3c\"/\"default-dockercfg-k7sph\"" Nov 02 08:31:13.619220 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.619150 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr"] Nov 02 08:31:13.627654 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.627627 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-12df3c\"/\"openshift-service-ca.crt\"" Nov 02 08:31:13.627654 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.627637 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-12df3c\"/\"kube-root-ca.crt\"" Nov 02 08:31:13.629069 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.629039 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:13.675397 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.675356 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-model-cache\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.675397 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.675399 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-dshm\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.675668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.675513 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-home\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.675668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.675549 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crndz\" (UniqueName: \"kubernetes.io/projected/345e41d9-05b3-4795-a9fc-fce423798f9d-kube-api-access-crndz\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.675783 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.675695 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-tmp-dir\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.675912 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.675884 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/345e41d9-05b3-4795-a9fc-fce423798f9d-tls-certs\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.700403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.700369 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-12df3c\"/\"rate-test-model-12df3c-kserve-self-signed-certs\"" Nov 02 08:31:13.768516 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.768477 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:13.777004 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.776969 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/345e41d9-05b3-4795-a9fc-fce423798f9d-tls-certs\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.777152 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.777038 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-model-cache\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.777152 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.777066 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-dshm\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.777152 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.777127 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-home\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.777344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.777176 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crndz\" (UniqueName: \"kubernetes.io/projected/345e41d9-05b3-4795-a9fc-fce423798f9d-kube-api-access-crndz\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.777344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.777283 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-tmp-dir\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.777643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.777617 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-home\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.777742 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.777718 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-tmp-dir\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.777821 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.777804 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-model-cache\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.779494 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.779466 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-dshm\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.779637 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.779618 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/345e41d9-05b3-4795-a9fc-fce423798f9d-tls-certs\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.799589 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.799552 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crndz\" (UniqueName: \"kubernetes.io/projected/345e41d9-05b3-4795-a9fc-fce423798f9d-kube-api-access-crndz\") pod \"rate-test-model-12df3c-kserve-79d87cb76f-5zbrr\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:13.913688 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:13.913595 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:14.229464 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:14.229431 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr"] Nov 02 08:31:14.231087 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:31:14.231053 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod345e41d9_05b3_4795_a9fc_fce423798f9d.slice/crio-8b248fb8d39b1ca99bb065c6950f4c15a5a20e11cfd9ea334b85e7817154d8f6 WatchSource:0}: Error finding container 8b248fb8d39b1ca99bb065c6950f4c15a5a20e11cfd9ea334b85e7817154d8f6: Status 404 returned error can't find the container with id 8b248fb8d39b1ca99bb065c6950f4c15a5a20e11cfd9ea334b85e7817154d8f6 Nov 02 08:31:14.776353 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:14.776317 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" event={"ID":"345e41d9-05b3-4795-a9fc-fce423798f9d","Type":"ContainerStarted","Data":"cbd930a5c62798d634223a6c7b9a073bf91a1435c521093824eadd273ab3f0d5"} Nov 02 08:31:14.776560 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:14.776363 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" event={"ID":"345e41d9-05b3-4795-a9fc-fce423798f9d","Type":"ContainerStarted","Data":"8b248fb8d39b1ca99bb065c6950f4c15a5a20e11cfd9ea334b85e7817154d8f6"} Nov 02 08:31:14.818154 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:14.818099 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" podStartSLOduration=1.818084671 podStartE2EDuration="1.818084671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:31:14.816868769 +0000 UTC m=+3975.115149608" watchObservedRunningTime="2026-11-02 08:31:14.818084671 +0000 UTC m=+3975.116365528" Nov 02 08:31:14.914637 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:14.914599 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:15.737723 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:15.737673 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:15.881521 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:15.881477 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:16.048542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:16.048453 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:17.987780 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:17.987716 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:18.646847 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:18.646808 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:19.688441 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:19.688409 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:21.346277 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:21.346233 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:23.204856 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:23.204816 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:23.913856 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:23.913820 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:23.926858 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:23.926824 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:24.845006 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:24.844969 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:24.850278 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:24.850249 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:31:26.102753 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:26.102708 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:26.617391 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:26.617356 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:27.965867 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:27.965828 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6"] Nov 02 08:31:27.966444 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:27.966068 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" podUID="9a39cd06-3ed4-4b12-9a75-f0cd30e4160e" containerName="maas-api" containerID="cri-o://05ac40eca641767a93ea1d4cee6e50278de01385697f81f4005c85bf83ab39d6" gracePeriod=30 Nov 02 08:31:28.056058 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.056019 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:28.260201 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.260146 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 08:31:28.359357 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.359323 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-openshift-service-ca\") pod \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " Nov 02 08:31:28.359563 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.359383 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-metrics-certs\") pod \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " Nov 02 08:31:28.359563 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.359421 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-kube-api-access-nxsb7\" (UniqueName: \"kubernetes.io/projected/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-kube-api-access-nxsb7\") pod \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " Nov 02 08:31:28.359563 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.359462 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-maas-api-tls\") pod \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\" (UID: \"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e\") " Nov 02 08:31:28.359743 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.359727 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-openshift-service-ca" pod "9a39cd06-3ed4-4b12-9a75-f0cd30e4160e" (UID: "9a39cd06-3ed4-4b12-9a75-f0cd30e4160e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:31:28.360099 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.360065 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:28.362835 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.362503 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-maas-api-tls" pod "9a39cd06-3ed4-4b12-9a75-f0cd30e4160e" (UID: "9a39cd06-3ed4-4b12-9a75-f0cd30e4160e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:31:28.363627 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.363596 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-kube-api-access-nxsb7" pod "9a39cd06-3ed4-4b12-9a75-f0cd30e4160e" (UID: "9a39cd06-3ed4-4b12-9a75-f0cd30e4160e"). InnerVolumeSpecName "kube-api-access-nxsb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:31:28.368007 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.367335 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-metrics-certs" pod "9a39cd06-3ed4-4b12-9a75-f0cd30e4160e" (UID: "9a39cd06-3ed4-4b12-9a75-f0cd30e4160e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:31:28.461314 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.461270 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:28.461314 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.461306 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxsb7\" (UniqueName: \"kubernetes.io/projected/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-kube-api-access-nxsb7\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:28.461314 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.461316 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:28.866684 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.866646 5899 generic.go:356] "Generic (PLEG): container finished" podID="9a39cd06-3ed4-4b12-9a75-f0cd30e4160e" containerID="05ac40eca641767a93ea1d4cee6e50278de01385697f81f4005c85bf83ab39d6" exitCode=0 Nov 02 08:31:28.866885 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.866692 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" Nov 02 08:31:28.866885 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.866703 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" event={"ID":"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e","Type":"ContainerDied","Data":"05ac40eca641767a93ea1d4cee6e50278de01385697f81f4005c85bf83ab39d6"} Nov 02 08:31:28.866885 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.866759 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6" event={"ID":"9a39cd06-3ed4-4b12-9a75-f0cd30e4160e","Type":"ContainerDied","Data":"c256ecfc04e0a27f1af7fb6f3593646d456aa8a350172b28de6f41fd445ddb8d"} Nov 02 08:31:28.866885 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.866785 5899 scope.go:122] "RemoveContainer" containerID="05ac40eca641767a93ea1d4cee6e50278de01385697f81f4005c85bf83ab39d6" Nov 02 08:31:28.877901 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.877880 5899 scope.go:122] "RemoveContainer" containerID="05ac40eca641767a93ea1d4cee6e50278de01385697f81f4005c85bf83ab39d6" Nov 02 08:31:28.878219 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:31:28.878193 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ac40eca641767a93ea1d4cee6e50278de01385697f81f4005c85bf83ab39d6\": container with ID starting with 05ac40eca641767a93ea1d4cee6e50278de01385697f81f4005c85bf83ab39d6 not found: ID does not exist" containerID="05ac40eca641767a93ea1d4cee6e50278de01385697f81f4005c85bf83ab39d6" Nov 02 08:31:28.878286 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.878232 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ac40eca641767a93ea1d4cee6e50278de01385697f81f4005c85bf83ab39d6"} err="failed to get container status \"05ac40eca641767a93ea1d4cee6e50278de01385697f81f4005c85bf83ab39d6\": rpc error: code = NotFound desc = could not find container \"05ac40eca641767a93ea1d4cee6e50278de01385697f81f4005c85bf83ab39d6\": container with ID starting with 05ac40eca641767a93ea1d4cee6e50278de01385697f81f4005c85bf83ab39d6 not found: ID does not exist" Nov 02 08:31:28.893358 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.893332 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:28.917384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.917343 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6"] Nov 02 08:31:28.927700 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:28.927665 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3575b9-7d7db78fcf-h2kz6"] Nov 02 08:31:29.823497 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:29.823458 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a39cd06-3ed4-4b12-9a75-f0cd30e4160e" path="/var/lib/kubelet/pods/9a39cd06-3ed4-4b12-9a75-f0cd30e4160e/volumes" Nov 02 08:31:30.703379 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:30.703330 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:32.706069 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:32.706014 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:32.920691 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:32.920651 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:33.547752 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:33.547719 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:33.974362 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:33.974266 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:34.497905 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:34.497864 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:34.641629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:34.641584 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:36.561692 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:36.561647 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:38.282042 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:38.282006 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:39.516796 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:39.516737 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:39.978562 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:39.978527 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:40.127967 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:40.127929 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:41.824721 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:41.824686 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:42.506554 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:42.506522 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:43.142008 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:31:43.141969 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:31:43.381325 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.381287 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn"] Nov 02 08:31:43.381593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.381536 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" podUID="df4e434e-62cc-4239-8e4b-d5a319dd963f" containerName="maas-api" containerID="cri-o://7d74de8792a6173a2ee394bbf370a8e76fb92b39f5a5de007aa2248d07fd5eda" gracePeriod=30 Nov 02 08:31:43.469344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.469306 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:43.678100 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.678029 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 08:31:43.856081 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.856043 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-metrics-certs\") pod \"df4e434e-62cc-4239-8e4b-d5a319dd963f\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " Nov 02 08:31:43.856274 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.856096 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df4e434e-62cc-4239-8e4b-d5a319dd963f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4e434e-62cc-4239-8e4b-d5a319dd963f-openshift-service-ca\") pod \"df4e434e-62cc-4239-8e4b-d5a319dd963f\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " Nov 02 08:31:43.856274 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.856188 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-maas-api-tls\") pod \"df4e434e-62cc-4239-8e4b-d5a319dd963f\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " Nov 02 08:31:43.856274 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.856213 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4e434e-62cc-4239-8e4b-d5a319dd963f-kube-api-access-8bzcq\" (UniqueName: \"kubernetes.io/projected/df4e434e-62cc-4239-8e4b-d5a319dd963f-kube-api-access-8bzcq\") pod \"df4e434e-62cc-4239-8e4b-d5a319dd963f\" (UID: \"df4e434e-62cc-4239-8e4b-d5a319dd963f\") " Nov 02 08:31:43.856602 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.856571 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4e434e-62cc-4239-8e4b-d5a319dd963f-openshift-service-ca" pod "df4e434e-62cc-4239-8e4b-d5a319dd963f" (UID: "df4e434e-62cc-4239-8e4b-d5a319dd963f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:31:43.858508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.858479 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-metrics-certs" pod "df4e434e-62cc-4239-8e4b-d5a319dd963f" (UID: "df4e434e-62cc-4239-8e4b-d5a319dd963f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:31:43.858626 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.858512 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4e434e-62cc-4239-8e4b-d5a319dd963f-kube-api-access-8bzcq" pod "df4e434e-62cc-4239-8e4b-d5a319dd963f" (UID: "df4e434e-62cc-4239-8e4b-d5a319dd963f"). InnerVolumeSpecName "kube-api-access-8bzcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:31:43.858856 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.858826 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-maas-api-tls" pod "df4e434e-62cc-4239-8e4b-d5a319dd963f" (UID: "df4e434e-62cc-4239-8e4b-d5a319dd963f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:31:43.957710 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.957611 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:43.957710 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.957642 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4e434e-62cc-4239-8e4b-d5a319dd963f-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:43.957710 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.957651 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4e434e-62cc-4239-8e4b-d5a319dd963f-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:43.957710 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.957660 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bzcq\" (UniqueName: \"kubernetes.io/projected/df4e434e-62cc-4239-8e4b-d5a319dd963f-kube-api-access-8bzcq\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:43.968032 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.968001 5899 generic.go:356] "Generic (PLEG): container finished" podID="df4e434e-62cc-4239-8e4b-d5a319dd963f" containerID="7d74de8792a6173a2ee394bbf370a8e76fb92b39f5a5de007aa2248d07fd5eda" exitCode=0 Nov 02 08:31:43.968227 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.968067 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" Nov 02 08:31:43.968227 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.968085 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" event={"ID":"df4e434e-62cc-4239-8e4b-d5a319dd963f","Type":"ContainerDied","Data":"7d74de8792a6173a2ee394bbf370a8e76fb92b39f5a5de007aa2248d07fd5eda"} Nov 02 08:31:43.968227 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.968145 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn" event={"ID":"df4e434e-62cc-4239-8e4b-d5a319dd963f","Type":"ContainerDied","Data":"d68dde38127d2566cfffb3242ab35b9419c57699a456c2512531f30813597290"} Nov 02 08:31:43.968227 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.968189 5899 scope.go:122] "RemoveContainer" containerID="7d74de8792a6173a2ee394bbf370a8e76fb92b39f5a5de007aa2248d07fd5eda" Nov 02 08:31:43.975585 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.975555 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm"] Nov 02 08:31:43.976097 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.976082 5899 cpu_manager.go:414] "containerMap: removing container" podUID="9a39cd06-3ed4-4b12-9a75-f0cd30e4160e" containerName="maas-api" Nov 02 08:31:43.976097 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.976097 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a39cd06-3ed4-4b12-9a75-f0cd30e4160e" containerName="maas-api" Nov 02 08:31:43.976275 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.976116 5899 cpu_manager.go:414] "containerMap: removing container" podUID="df4e434e-62cc-4239-8e4b-d5a319dd963f" containerName="maas-api" Nov 02 08:31:43.976275 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.976121 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4e434e-62cc-4239-8e4b-d5a319dd963f" containerName="maas-api" Nov 02 08:31:43.976275 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.976253 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a39cd06-3ed4-4b12-9a75-f0cd30e4160e" containerName="maas-api" Nov 02 08:31:43.976275 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.976264 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4e434e-62cc-4239-8e4b-d5a319dd963f" containerName="maas-api" Nov 02 08:31:43.979427 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.979406 5899 scope.go:122] "RemoveContainer" containerID="7d74de8792a6173a2ee394bbf370a8e76fb92b39f5a5de007aa2248d07fd5eda" Nov 02 08:31:43.979723 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:31:43.979704 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d74de8792a6173a2ee394bbf370a8e76fb92b39f5a5de007aa2248d07fd5eda\": container with ID starting with 7d74de8792a6173a2ee394bbf370a8e76fb92b39f5a5de007aa2248d07fd5eda not found: ID does not exist" containerID="7d74de8792a6173a2ee394bbf370a8e76fb92b39f5a5de007aa2248d07fd5eda" Nov 02 08:31:43.979795 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.979738 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d74de8792a6173a2ee394bbf370a8e76fb92b39f5a5de007aa2248d07fd5eda"} err="failed to get container status \"7d74de8792a6173a2ee394bbf370a8e76fb92b39f5a5de007aa2248d07fd5eda\": rpc error: code = NotFound desc = could not find container \"7d74de8792a6173a2ee394bbf370a8e76fb92b39f5a5de007aa2248d07fd5eda\": container with ID starting with 7d74de8792a6173a2ee394bbf370a8e76fb92b39f5a5de007aa2248d07fd5eda not found: ID does not exist" Nov 02 08:31:43.981106 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.981090 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:43.996775 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.996754 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-3d8293-serving-cert\"" Nov 02 08:31:43.999960 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:43.999931 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm"] Nov 02 08:31:44.022450 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.022410 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn"] Nov 02 08:31:44.027645 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.027615 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e4c131-57bf54d575-69ddn"] Nov 02 08:31:44.094338 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.094307 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:44.159398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.159359 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a71828-ac01-499e-90d3-6bc754414014-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:44.159398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.159411 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:44.159925 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.159550 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbn2k\" (UniqueName: \"kubernetes.io/projected/f3a71828-ac01-499e-90d3-6bc754414014-kube-api-access-zbn2k\") pod \"maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:44.159925 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.159640 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:44.260727 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.260688 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a71828-ac01-499e-90d3-6bc754414014-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:44.260727 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.260734 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:44.260966 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.260774 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbn2k\" (UniqueName: \"kubernetes.io/projected/f3a71828-ac01-499e-90d3-6bc754414014-kube-api-access-zbn2k\") pod \"maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:44.260966 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.260820 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:44.261513 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.261485 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a71828-ac01-499e-90d3-6bc754414014-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:44.263311 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.263290 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:44.263400 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.263317 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:44.315877 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.315827 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbn2k\" (UniqueName: \"kubernetes.io/projected/f3a71828-ac01-499e-90d3-6bc754414014-kube-api-access-zbn2k\") pod \"maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:44.367320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.367286 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:44.596103 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.596008 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:44.901908 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.901871 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm"] Nov 02 08:31:44.903821 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:31:44.903788 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a71828_ac01_499e_90d3_6bc754414014.slice/crio-90593831659d35c2d96886a41ccd0d41125b393b063fe3980989903038a49ef8 WatchSource:0}: Error finding container 90593831659d35c2d96886a41ccd0d41125b393b063fe3980989903038a49ef8: Status 404 returned error can't find the container with id 90593831659d35c2d96886a41ccd0d41125b393b063fe3980989903038a49ef8 Nov 02 08:31:44.978997 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:44.978958 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" event={"ID":"f3a71828-ac01-499e-90d3-6bc754414014","Type":"ContainerStarted","Data":"90593831659d35c2d96886a41ccd0d41125b393b063fe3980989903038a49ef8"} Nov 02 08:31:45.335428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:45.335385 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:45.487673 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:45.487639 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:45.819989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:45.819942 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df4e434e-62cc-4239-8e4b-d5a319dd963f" path="/var/lib/kubelet/pods/df4e434e-62cc-4239-8e4b-d5a319dd963f/volumes" Nov 02 08:31:45.986896 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:45.986860 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" event={"ID":"f3a71828-ac01-499e-90d3-6bc754414014","Type":"ContainerStarted","Data":"aff3e890ec3ccd14129533981db0d030cbf3825864ab19720ed0c00386e4250f"} Nov 02 08:31:45.987073 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:45.986947 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:46.039292 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:46.039236 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" podStartSLOduration=2.6701388489999998 podStartE2EDuration="3.039220839s" totalImagesPullingTime="369.08199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:31:46.037101259 +0000 UTC m=+4006.335382087" watchObservedRunningTime="2026-11-02 08:31:46.039220839 +0000 UTC m=+4006.337501665" Nov 02 08:31:46.093819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:46.093731 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:46.242522 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:46.242482 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:46.638401 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:46.638364 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:47.371635 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:47.371590 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:47.730311 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:47.730271 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:47.865488 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:47.865457 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:48.158547 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:48.158430 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:49.369487 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:49.369448 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:50.110954 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:50.110915 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:51.935631 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:51.935569 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:51.998402 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:51.998372 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:31:53.430006 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.429967 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46"] Nov 02 08:31:53.430550 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.430259 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" podUID="1da87585-b423-4543-b5bd-7c3dc5e55063" containerName="maas-api" containerID="cri-o://980dddf1221353074379b66f914a3fefb97439721e48498e1ab80c2e7fadbe1d" gracePeriod=30 Nov 02 08:31:53.606978 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.606903 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:53.740082 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.740047 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 08:31:53.801180 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.797688 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da87585-b423-4543-b5bd-7c3dc5e55063-kube-api-access-vsw5n\" (UniqueName: \"kubernetes.io/projected/1da87585-b423-4543-b5bd-7c3dc5e55063-kube-api-access-vsw5n\") pod \"1da87585-b423-4543-b5bd-7c3dc5e55063\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " Nov 02 08:31:53.801180 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.797788 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-maas-api-tls\") pod \"1da87585-b423-4543-b5bd-7c3dc5e55063\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " Nov 02 08:31:53.801180 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.797826 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-metrics-certs\") pod \"1da87585-b423-4543-b5bd-7c3dc5e55063\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " Nov 02 08:31:53.801180 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.797956 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da87585-b423-4543-b5bd-7c3dc5e55063-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da87585-b423-4543-b5bd-7c3dc5e55063-openshift-service-ca\") pod \"1da87585-b423-4543-b5bd-7c3dc5e55063\" (UID: \"1da87585-b423-4543-b5bd-7c3dc5e55063\") " Nov 02 08:31:53.801180 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.798680 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da87585-b423-4543-b5bd-7c3dc5e55063-openshift-service-ca" pod "1da87585-b423-4543-b5bd-7c3dc5e55063" (UID: "1da87585-b423-4543-b5bd-7c3dc5e55063"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:31:53.801904 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.801874 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da87585-b423-4543-b5bd-7c3dc5e55063-kube-api-access-vsw5n" pod "1da87585-b423-4543-b5bd-7c3dc5e55063" (UID: "1da87585-b423-4543-b5bd-7c3dc5e55063"). InnerVolumeSpecName "kube-api-access-vsw5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:31:53.804785 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.804746 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-maas-api-tls" pod "1da87585-b423-4543-b5bd-7c3dc5e55063" (UID: "1da87585-b423-4543-b5bd-7c3dc5e55063"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:31:53.808337 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.808306 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-metrics-certs" pod "1da87585-b423-4543-b5bd-7c3dc5e55063" (UID: "1da87585-b423-4543-b5bd-7c3dc5e55063"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:31:53.818654 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.818337 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:53.900178 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.900119 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsw5n\" (UniqueName: \"kubernetes.io/projected/1da87585-b423-4543-b5bd-7c3dc5e55063-kube-api-access-vsw5n\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:53.900178 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.900176 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:53.900409 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.900192 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da87585-b423-4543-b5bd-7c3dc5e55063-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:53.900409 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.900204 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da87585-b423-4543-b5bd-7c3dc5e55063-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:31:53.962108 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:53.962023 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:54.039954 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:54.039916 5899 generic.go:356] "Generic (PLEG): container finished" podID="1da87585-b423-4543-b5bd-7c3dc5e55063" containerID="980dddf1221353074379b66f914a3fefb97439721e48498e1ab80c2e7fadbe1d" exitCode=0 Nov 02 08:31:54.040202 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:54.039971 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" Nov 02 08:31:54.040202 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:54.039999 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" event={"ID":"1da87585-b423-4543-b5bd-7c3dc5e55063","Type":"ContainerDied","Data":"980dddf1221353074379b66f914a3fefb97439721e48498e1ab80c2e7fadbe1d"} Nov 02 08:31:54.040202 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:54.040072 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46" event={"ID":"1da87585-b423-4543-b5bd-7c3dc5e55063","Type":"ContainerDied","Data":"7c1fdae4de814606d813193dee9f1207942f25cd7b0ea2c013f141ccdfdada36"} Nov 02 08:31:54.040202 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:54.040104 5899 scope.go:122] "RemoveContainer" containerID="980dddf1221353074379b66f914a3fefb97439721e48498e1ab80c2e7fadbe1d" Nov 02 08:31:54.053859 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:54.053824 5899 scope.go:122] "RemoveContainer" containerID="980dddf1221353074379b66f914a3fefb97439721e48498e1ab80c2e7fadbe1d" Nov 02 08:31:54.055216 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:31:54.054422 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980dddf1221353074379b66f914a3fefb97439721e48498e1ab80c2e7fadbe1d\": container with ID starting with 980dddf1221353074379b66f914a3fefb97439721e48498e1ab80c2e7fadbe1d not found: ID does not exist" containerID="980dddf1221353074379b66f914a3fefb97439721e48498e1ab80c2e7fadbe1d" Nov 02 08:31:54.055216 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:54.054463 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980dddf1221353074379b66f914a3fefb97439721e48498e1ab80c2e7fadbe1d"} err="failed to get container status \"980dddf1221353074379b66f914a3fefb97439721e48498e1ab80c2e7fadbe1d\": rpc error: code = NotFound desc = could not find container \"980dddf1221353074379b66f914a3fefb97439721e48498e1ab80c2e7fadbe1d\": container with ID starting with 980dddf1221353074379b66f914a3fefb97439721e48498e1ab80c2e7fadbe1d not found: ID does not exist" Nov 02 08:31:54.097973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:54.097938 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46"] Nov 02 08:31:54.105829 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:54.105791 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6929a4-66495749d6-jkc46"] Nov 02 08:31:54.355607 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:54.355561 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:54.566477 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:54.566445 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:55.820642 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:55.820607 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da87585-b423-4543-b5bd-7c3dc5e55063" path="/var/lib/kubelet/pods/1da87585-b423-4543-b5bd-7c3dc5e55063/volumes" Nov 02 08:31:56.198022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:56.197948 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:56.623593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:56.623560 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:58.020458 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:58.020420 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:31:59.973371 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:31:59.973331 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:00.567052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:00.567016 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:00.729824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:00.729790 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:01.395334 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:01.395253 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:01.770240 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:01.770203 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:01.920435 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:01.920398 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:02.196633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:02.196554 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:03.514545 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:03.514501 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:03.814237 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:03.814142 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:05.631397 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:05.631351 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:07.129388 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:07.129351 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl"] Nov 02 08:32:07.311411 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:07.311373 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:07.466420 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:07.466342 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:08.011882 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:08.011844 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:09.210715 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:09.210673 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:09.352553 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:09.352513 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:10.843176 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:10.843125 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:12.618420 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:12.618385 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:13.142339 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:32:13.142299 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:32:13.971755 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:13.971716 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:14.111608 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:14.111569 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:14.681093 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:14.681055 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:15.327960 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:15.327923 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:16.662324 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:16.662285 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:16.835366 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:16.835334 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:17.066436 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:17.066396 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:18.413779 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.413735 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk"] Nov 02 08:32:18.414321 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.414003 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" podUID="596b979e-253a-4286-88ca-e0f737d3be85" containerName="maas-api" containerID="cri-o://415e46d5b4299f667ddcf931e46bf2f1ceae7e8b6ee6e42d1b22449cdc6bd31b" gracePeriod=30 Nov 02 08:32:18.698648 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.698617 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:18.726935 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.726903 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 08:32:18.834889 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.834847 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-maas-api-tls\") pod \"596b979e-253a-4286-88ca-e0f737d3be85\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " Nov 02 08:32:18.835081 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.834998 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-metrics-certs\" (UniqueName: \"kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-metrics-certs\") pod \"596b979e-253a-4286-88ca-e0f737d3be85\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " Nov 02 08:32:18.835081 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.835040 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596b979e-253a-4286-88ca-e0f737d3be85-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/596b979e-253a-4286-88ca-e0f737d3be85-openshift-service-ca\") pod \"596b979e-253a-4286-88ca-e0f737d3be85\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " Nov 02 08:32:18.835209 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.835125 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596b979e-253a-4286-88ca-e0f737d3be85-kube-api-access-kt6xj\" (UniqueName: \"kubernetes.io/projected/596b979e-253a-4286-88ca-e0f737d3be85-kube-api-access-kt6xj\") pod \"596b979e-253a-4286-88ca-e0f737d3be85\" (UID: \"596b979e-253a-4286-88ca-e0f737d3be85\") " Nov 02 08:32:18.835586 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.835557 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596b979e-253a-4286-88ca-e0f737d3be85-openshift-service-ca" pod "596b979e-253a-4286-88ca-e0f737d3be85" (UID: "596b979e-253a-4286-88ca-e0f737d3be85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:32:18.837224 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.837197 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-metrics-certs" pod "596b979e-253a-4286-88ca-e0f737d3be85" (UID: "596b979e-253a-4286-88ca-e0f737d3be85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:32:18.837442 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.837413 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596b979e-253a-4286-88ca-e0f737d3be85-kube-api-access-kt6xj" pod "596b979e-253a-4286-88ca-e0f737d3be85" (UID: "596b979e-253a-4286-88ca-e0f737d3be85"). InnerVolumeSpecName "kube-api-access-kt6xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:32:18.837559 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.837501 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-maas-api-tls" pod "596b979e-253a-4286-88ca-e0f737d3be85" (UID: "596b979e-253a-4286-88ca-e0f737d3be85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:32:18.936963 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.936926 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:18.936963 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.936957 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/596b979e-253a-4286-88ca-e0f737d3be85-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:18.937213 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.936980 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt6xj\" (UniqueName: \"kubernetes.io/projected/596b979e-253a-4286-88ca-e0f737d3be85-kube-api-access-kt6xj\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:18.937213 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:18.936992 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/596b979e-253a-4286-88ca-e0f737d3be85-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:19.221127 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:19.221093 5899 generic.go:356] "Generic (PLEG): container finished" podID="596b979e-253a-4286-88ca-e0f737d3be85" containerID="415e46d5b4299f667ddcf931e46bf2f1ceae7e8b6ee6e42d1b22449cdc6bd31b" exitCode=0 Nov 02 08:32:19.221319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:19.221142 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" Nov 02 08:32:19.221319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:19.221205 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" event={"ID":"596b979e-253a-4286-88ca-e0f737d3be85","Type":"ContainerDied","Data":"415e46d5b4299f667ddcf931e46bf2f1ceae7e8b6ee6e42d1b22449cdc6bd31b"} Nov 02 08:32:19.221319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:19.221260 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk" event={"ID":"596b979e-253a-4286-88ca-e0f737d3be85","Type":"ContainerDied","Data":"f7e118d9e9d01bf5f46db77f3985a5cd602c597f29f50955ded52ddcc268f9dd"} Nov 02 08:32:19.221319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:19.221289 5899 scope.go:122] "RemoveContainer" containerID="415e46d5b4299f667ddcf931e46bf2f1ceae7e8b6ee6e42d1b22449cdc6bd31b" Nov 02 08:32:19.231630 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:19.231610 5899 scope.go:122] "RemoveContainer" containerID="415e46d5b4299f667ddcf931e46bf2f1ceae7e8b6ee6e42d1b22449cdc6bd31b" Nov 02 08:32:19.231888 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:32:19.231868 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415e46d5b4299f667ddcf931e46bf2f1ceae7e8b6ee6e42d1b22449cdc6bd31b\": container with ID starting with 415e46d5b4299f667ddcf931e46bf2f1ceae7e8b6ee6e42d1b22449cdc6bd31b not found: ID does not exist" containerID="415e46d5b4299f667ddcf931e46bf2f1ceae7e8b6ee6e42d1b22449cdc6bd31b" Nov 02 08:32:19.231942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:19.231895 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415e46d5b4299f667ddcf931e46bf2f1ceae7e8b6ee6e42d1b22449cdc6bd31b"} err="failed to get container status \"415e46d5b4299f667ddcf931e46bf2f1ceae7e8b6ee6e42d1b22449cdc6bd31b\": rpc error: code = NotFound desc = could not find container \"415e46d5b4299f667ddcf931e46bf2f1ceae7e8b6ee6e42d1b22449cdc6bd31b\": container with ID starting with 415e46d5b4299f667ddcf931e46bf2f1ceae7e8b6ee6e42d1b22449cdc6bd31b not found: ID does not exist" Nov 02 08:32:19.271131 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:19.271088 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk"] Nov 02 08:32:19.282830 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:19.282792 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0fbc17-57469b95f5-m29lk"] Nov 02 08:32:19.366343 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:19.366312 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:19.820617 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:19.820581 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596b979e-253a-4286-88ca-e0f737d3be85" path="/var/lib/kubelet/pods/596b979e-253a-4286-88ca-e0f737d3be85/volumes" Nov 02 08:32:20.817527 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:20.817491 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:21.168575 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:21.168487 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:21.376610 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:21.376568 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:21.914008 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:21.913971 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:22.150609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.150567 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerName="main" containerID="cri-o://3c5c2b4d2dd2bdd898106d25a96b8b4697fe23eed9a2eb5fd283494885e6e187" gracePeriod=45 Nov 02 08:32:22.219064 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.219020 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:22.368298 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.367510 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:22.418080 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.418052 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 08:32:22.583075 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.583032 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tls-certs\") pod \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " Nov 02 08:32:22.583075 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.583073 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tmp-dir\") pod \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " Nov 02 08:32:22.583350 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.583137 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-kube-api-access-cln5l\" (UniqueName: \"kubernetes.io/projected/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-kube-api-access-cln5l\") pod \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " Nov 02 08:32:22.583350 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.583191 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-home\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-home\") pod \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " Nov 02 08:32:22.583350 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.583230 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-dshm\") pod \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " Nov 02 08:32:22.583350 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.583309 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-model-cache\") pod \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\" (UID: \"59c52c76-c947-49cc-bbf0-1ea2cf3834c5\") " Nov 02 08:32:22.583581 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.583424 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tmp-dir" pod "59c52c76-c947-49cc-bbf0-1ea2cf3834c5" (UID: "59c52c76-c947-49cc-bbf0-1ea2cf3834c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:32:22.583581 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.583517 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-home" pod "59c52c76-c947-49cc-bbf0-1ea2cf3834c5" (UID: "59c52c76-c947-49cc-bbf0-1ea2cf3834c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:32:22.583813 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.583785 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-model-cache" pod "59c52c76-c947-49cc-bbf0-1ea2cf3834c5" (UID: "59c52c76-c947-49cc-bbf0-1ea2cf3834c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:32:22.584225 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.584200 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:22.584353 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.584256 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:22.584353 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.584270 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:22.585668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.585636 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-kube-api-access-cln5l" pod "59c52c76-c947-49cc-bbf0-1ea2cf3834c5" (UID: "59c52c76-c947-49cc-bbf0-1ea2cf3834c5"). InnerVolumeSpecName "kube-api-access-cln5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:32:22.585942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.585918 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tls-certs" pod "59c52c76-c947-49cc-bbf0-1ea2cf3834c5" (UID: "59c52c76-c947-49cc-bbf0-1ea2cf3834c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:32:22.586153 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.586126 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-dshm" pod "59c52c76-c947-49cc-bbf0-1ea2cf3834c5" (UID: "59c52c76-c947-49cc-bbf0-1ea2cf3834c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:32:22.685535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.685502 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:22.685535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.685531 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cln5l\" (UniqueName: \"kubernetes.io/projected/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-kube-api-access-cln5l\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:22.685535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:22.685541 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c52c76-c947-49cc-bbf0-1ea2cf3834c5-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:23.238236 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:23.238194 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:23.254460 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:23.254419 5899 generic.go:356] "Generic (PLEG): container finished" podID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerID="3c5c2b4d2dd2bdd898106d25a96b8b4697fe23eed9a2eb5fd283494885e6e187" exitCode=0 Nov 02 08:32:23.254460 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:23.254473 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" Nov 02 08:32:23.254802 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:23.254629 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" event={"ID":"59c52c76-c947-49cc-bbf0-1ea2cf3834c5","Type":"ContainerDied","Data":"3c5c2b4d2dd2bdd898106d25a96b8b4697fe23eed9a2eb5fd283494885e6e187"} Nov 02 08:32:23.254802 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:23.254665 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl" event={"ID":"59c52c76-c947-49cc-bbf0-1ea2cf3834c5","Type":"ContainerDied","Data":"e49b237ddcd32fa5e833622d4cec933a7d4b1496461a6421a0ad59c08fb5cd9b"} Nov 02 08:32:23.254802 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:23.254688 5899 scope.go:122] "RemoveContainer" containerID="3c5c2b4d2dd2bdd898106d25a96b8b4697fe23eed9a2eb5fd283494885e6e187" Nov 02 08:32:23.265663 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:23.265640 5899 scope.go:122] "RemoveContainer" containerID="3c5c2b4d2dd2bdd898106d25a96b8b4697fe23eed9a2eb5fd283494885e6e187" Nov 02 08:32:23.265987 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:32:23.265965 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5c2b4d2dd2bdd898106d25a96b8b4697fe23eed9a2eb5fd283494885e6e187\": container with ID starting with 3c5c2b4d2dd2bdd898106d25a96b8b4697fe23eed9a2eb5fd283494885e6e187 not found: ID does not exist" containerID="3c5c2b4d2dd2bdd898106d25a96b8b4697fe23eed9a2eb5fd283494885e6e187" Nov 02 08:32:23.266071 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:23.266002 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5c2b4d2dd2bdd898106d25a96b8b4697fe23eed9a2eb5fd283494885e6e187"} err="failed to get container status \"3c5c2b4d2dd2bdd898106d25a96b8b4697fe23eed9a2eb5fd283494885e6e187\": rpc error: code = NotFound desc = could not find container \"3c5c2b4d2dd2bdd898106d25a96b8b4697fe23eed9a2eb5fd283494885e6e187\": container with ID starting with 3c5c2b4d2dd2bdd898106d25a96b8b4697fe23eed9a2eb5fd283494885e6e187 not found: ID does not exist" Nov 02 08:32:23.313926 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:23.313885 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl"] Nov 02 08:32:23.317274 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:23.317246 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7e7a83/distinct-w1-7e7a83-kserve-86f4b54c5c-nrrfl"] Nov 02 08:32:23.820541 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:23.820501 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" path="/var/lib/kubelet/pods/59c52c76-c947-49cc-bbf0-1ea2cf3834c5/volumes" Nov 02 08:32:24.107523 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:24.107431 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:24.535660 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:24.535626 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:25.091116 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:25.091067 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:25.264392 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:25.264361 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:26.738742 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:26.738696 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:26.899511 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:26.899470 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:27.058144 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:27.058055 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:28.432401 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:28.432362 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:28.635260 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:28.635222 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:29.003481 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:29.003444 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:30.238340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:30.238301 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:30.381366 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:30.381335 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:30.615052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:30.614963 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:30.948466 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:30.948376 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr"] Nov 02 08:32:30.948646 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:30.948622 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" podUID="da7b8863-0ace-444b-ab7d-2220f7f3137a" containerName="maas-api" containerID="cri-o://ff8025d780f2fc04f036e367547bd4c2c9136f5bea0c3e31975c50fb9b3f8b41" gracePeriod=30 Nov 02 08:32:31.004962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.004923 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:31.256508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.256480 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 08:32:31.282539 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.282508 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-maas-api-tls\") pod \"da7b8863-0ace-444b-ab7d-2220f7f3137a\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " Nov 02 08:32:31.282703 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.282604 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7b8863-0ace-444b-ab7d-2220f7f3137a-kube-api-access-hrnz7\" (UniqueName: \"kubernetes.io/projected/da7b8863-0ace-444b-ab7d-2220f7f3137a-kube-api-access-hrnz7\") pod \"da7b8863-0ace-444b-ab7d-2220f7f3137a\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " Nov 02 08:32:31.282703 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.282668 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-metrics-certs\") pod \"da7b8863-0ace-444b-ab7d-2220f7f3137a\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " Nov 02 08:32:31.282850 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.282799 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7b8863-0ace-444b-ab7d-2220f7f3137a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7b8863-0ace-444b-ab7d-2220f7f3137a-openshift-service-ca\") pod \"da7b8863-0ace-444b-ab7d-2220f7f3137a\" (UID: \"da7b8863-0ace-444b-ab7d-2220f7f3137a\") " Nov 02 08:32:31.283213 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.283153 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7b8863-0ace-444b-ab7d-2220f7f3137a-openshift-service-ca" pod "da7b8863-0ace-444b-ab7d-2220f7f3137a" (UID: "da7b8863-0ace-444b-ab7d-2220f7f3137a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:32:31.284750 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.284728 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-maas-api-tls" pod "da7b8863-0ace-444b-ab7d-2220f7f3137a" (UID: "da7b8863-0ace-444b-ab7d-2220f7f3137a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:32:31.285198 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.285151 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-metrics-certs" pod "da7b8863-0ace-444b-ab7d-2220f7f3137a" (UID: "da7b8863-0ace-444b-ab7d-2220f7f3137a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:32:31.285251 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.285220 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7b8863-0ace-444b-ab7d-2220f7f3137a-kube-api-access-hrnz7" pod "da7b8863-0ace-444b-ab7d-2220f7f3137a" (UID: "da7b8863-0ace-444b-ab7d-2220f7f3137a"). InnerVolumeSpecName "kube-api-access-hrnz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:32:31.306250 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.306131 5899 generic.go:356] "Generic (PLEG): container finished" podID="da7b8863-0ace-444b-ab7d-2220f7f3137a" containerID="ff8025d780f2fc04f036e367547bd4c2c9136f5bea0c3e31975c50fb9b3f8b41" exitCode=0 Nov 02 08:32:31.306714 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.306634 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" event={"ID":"da7b8863-0ace-444b-ab7d-2220f7f3137a","Type":"ContainerDied","Data":"ff8025d780f2fc04f036e367547bd4c2c9136f5bea0c3e31975c50fb9b3f8b41"} Nov 02 08:32:31.310633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.306856 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" event={"ID":"da7b8863-0ace-444b-ab7d-2220f7f3137a","Type":"ContainerDied","Data":"0f70d8771ea4b97a21a1deadad28f7c5fdccd12a1f3b04407777a8864a368a82"} Nov 02 08:32:31.310633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.306898 5899 scope.go:122] "RemoveContainer" containerID="ff8025d780f2fc04f036e367547bd4c2c9136f5bea0c3e31975c50fb9b3f8b41" Nov 02 08:32:31.310633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.307051 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr" Nov 02 08:32:31.331685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.331659 5899 scope.go:122] "RemoveContainer" containerID="ff8025d780f2fc04f036e367547bd4c2c9136f5bea0c3e31975c50fb9b3f8b41" Nov 02 08:32:31.332066 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:32:31.332034 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8025d780f2fc04f036e367547bd4c2c9136f5bea0c3e31975c50fb9b3f8b41\": container with ID starting with ff8025d780f2fc04f036e367547bd4c2c9136f5bea0c3e31975c50fb9b3f8b41 not found: ID does not exist" containerID="ff8025d780f2fc04f036e367547bd4c2c9136f5bea0c3e31975c50fb9b3f8b41" Nov 02 08:32:31.332243 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.332069 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8025d780f2fc04f036e367547bd4c2c9136f5bea0c3e31975c50fb9b3f8b41"} err="failed to get container status \"ff8025d780f2fc04f036e367547bd4c2c9136f5bea0c3e31975c50fb9b3f8b41\": rpc error: code = NotFound desc = could not find container \"ff8025d780f2fc04f036e367547bd4c2c9136f5bea0c3e31975c50fb9b3f8b41\": container with ID starting with ff8025d780f2fc04f036e367547bd4c2c9136f5bea0c3e31975c50fb9b3f8b41 not found: ID does not exist" Nov 02 08:32:31.364382 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.364348 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr"] Nov 02 08:32:31.366835 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.366808 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7e7a83-58fcf874b6-mwrtr"] Nov 02 08:32:31.384235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.384208 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7b8863-0ace-444b-ab7d-2220f7f3137a-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:31.384235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.384233 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:31.384385 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.384246 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrnz7\" (UniqueName: \"kubernetes.io/projected/da7b8863-0ace-444b-ab7d-2220f7f3137a-kube-api-access-hrnz7\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:31.384385 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.384255 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7b8863-0ace-444b-ab7d-2220f7f3137a-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:32:31.820747 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.820704 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7b8863-0ace-444b-ab7d-2220f7f3137a" path="/var/lib/kubelet/pods/da7b8863-0ace-444b-ab7d-2220f7f3137a/volumes" Nov 02 08:32:31.833186 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:31.833127 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:33.687716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:33.687683 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:35.679902 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:35.679859 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:37.199974 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:37.199940 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:37.456452 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:37.456368 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:37.723529 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:37.723439 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:39.525791 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:39.525750 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:41.324713 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:41.324609 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:43.140913 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:32:43.140878 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:32:43.190950 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:43.190916 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:45.160668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:45.160631 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:46.185184 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:46.184626 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:46.358588 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:46.358540 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:46.826675 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:46.826644 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:47.429806 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:47.429745 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:47.721935 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:47.721816 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:49.537971 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:49.537934 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:49.799985 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:49.799887 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:51.626583 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:51.626531 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:53.346451 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:53.346417 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:54.949009 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:54.948976 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:55.099675 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:55.099644 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:55.380719 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:55.380685 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:55.531391 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:55.531353 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:56.855622 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:56.855586 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:58.450299 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.450263 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb"] Nov 02 08:32:58.450806 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.450791 5899 cpu_manager.go:414] "containerMap: removing container" podUID="596b979e-253a-4286-88ca-e0f737d3be85" containerName="maas-api" Nov 02 08:32:58.450865 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.450806 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596b979e-253a-4286-88ca-e0f737d3be85" containerName="maas-api" Nov 02 08:32:58.450865 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.450818 5899 cpu_manager.go:414] "containerMap: removing container" podUID="da7b8863-0ace-444b-ab7d-2220f7f3137a" containerName="maas-api" Nov 02 08:32:58.450865 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.450823 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7b8863-0ace-444b-ab7d-2220f7f3137a" containerName="maas-api" Nov 02 08:32:58.450865 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.450856 5899 cpu_manager.go:414] "containerMap: removing container" podUID="1da87585-b423-4543-b5bd-7c3dc5e55063" containerName="maas-api" Nov 02 08:32:58.450865 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.450861 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da87585-b423-4543-b5bd-7c3dc5e55063" containerName="maas-api" Nov 02 08:32:58.451022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.450877 5899 cpu_manager.go:414] "containerMap: removing container" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerName="main" Nov 02 08:32:58.451022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.450881 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerName="main" Nov 02 08:32:58.451022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.450976 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c52c76-c947-49cc-bbf0-1ea2cf3834c5" containerName="main" Nov 02 08:32:58.451022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.450987 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="596b979e-253a-4286-88ca-e0f737d3be85" containerName="maas-api" Nov 02 08:32:58.451022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.450995 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7b8863-0ace-444b-ab7d-2220f7f3137a" containerName="maas-api" Nov 02 08:32:58.451022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.451005 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da87585-b423-4543-b5bd-7c3dc5e55063" containerName="maas-api" Nov 02 08:32:58.454567 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.454546 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:58.479746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.479716 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-a44352-serving-cert\"" Nov 02 08:32:58.483369 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.483336 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb"] Nov 02 08:32:58.507935 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.507901 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e264650-6da9-478e-b892-45afa95918e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:58.508074 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.507941 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbpqp\" (UniqueName: \"kubernetes.io/projected/9e264650-6da9-478e-b892-45afa95918e9-kube-api-access-mbpqp\") pod \"maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:58.508074 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.508024 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:58.508200 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.508098 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:58.568372 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.568334 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:58.609147 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.609113 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e264650-6da9-478e-b892-45afa95918e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:58.609334 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.609153 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbpqp\" (UniqueName: \"kubernetes.io/projected/9e264650-6da9-478e-b892-45afa95918e9-kube-api-access-mbpqp\") pod \"maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:58.609334 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.609215 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:58.609334 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.609266 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:58.609469 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:32:58.609401 5899 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-serving-cert: secret "maas-api-e2e-shared-b-w2-a44352-serving-cert" not found Nov 02 08:32:58.609469 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:32:58.609457 5899 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-maas-api-tls podName:9e264650-6da9-478e-b892-45afa95918e9 nodeName:}" failed. No retries permitted until 2026-11-02 08:32:59.109442074 +0000 UTC m=+4079.407722890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-maas-api-tls") pod "maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" (UID: "9e264650-6da9-478e-b892-45afa95918e9") : secret "maas-api-e2e-shared-b-w2-a44352-serving-cert" not found Nov 02 08:32:58.609891 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.609868 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e264650-6da9-478e-b892-45afa95918e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:58.611655 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.611635 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:58.634009 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.633974 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbpqp\" (UniqueName: \"kubernetes.io/projected/9e264650-6da9-478e-b892-45afa95918e9-kube-api-access-mbpqp\") pod \"maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:58.703248 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.703146 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:58.843333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:58.843302 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:32:59.115636 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:59.115599 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:59.118142 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:59.118114 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:59.368148 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:59.368056 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:32:59.679023 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:32:59.678997 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb"] Nov 02 08:32:59.681816 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:32:59.681790 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e264650_6da9_478e_b892_45afa95918e9.slice/crio-28867b7668d25a7bd7b43e8ecd0334b75fb95cf02b61dcf3c6c41bdea64a7fb4 WatchSource:0}: Error finding container 28867b7668d25a7bd7b43e8ecd0334b75fb95cf02b61dcf3c6c41bdea64a7fb4: Status 404 returned error can't find the container with id 28867b7668d25a7bd7b43e8ecd0334b75fb95cf02b61dcf3c6c41bdea64a7fb4 Nov 02 08:33:00.498815 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:00.498774 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" event={"ID":"9e264650-6da9-478e-b892-45afa95918e9","Type":"ContainerStarted","Data":"c93e4dd27efe3d09e0e0aaeac3db5d8717c34f528b0322f6aaacdf2fe01c6906"} Nov 02 08:33:00.499008 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:00.498840 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" event={"ID":"9e264650-6da9-478e-b892-45afa95918e9","Type":"ContainerStarted","Data":"28867b7668d25a7bd7b43e8ecd0334b75fb95cf02b61dcf3c6c41bdea64a7fb4"} Nov 02 08:33:00.499008 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:00.498982 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:33:00.521440 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:00.521410 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:00.544606 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:00.544535 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" podStartSLOduration=2.140753382 podStartE2EDuration="2.544515795s" totalImagesPullingTime="403.762413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:33:00.541311647 +0000 UTC m=+4080.839592497" watchObservedRunningTime="2026-11-02 08:33:00.544515795 +0000 UTC m=+4080.842796621" Nov 02 08:33:02.373959 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:02.373930 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:04.068405 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:04.068371 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:05.825328 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:05.825284 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:06.509239 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:06.509207 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:33:07.715886 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:07.715779 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:09.486425 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:09.486389 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:09.654137 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:09.654101 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:11.216475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:11.216437 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:13.032719 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:13.032680 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:13.141674 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:33:13.141638 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:33:14.958654 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:14.958610 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:16.797578 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:16.797541 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:18.651441 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:18.651404 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:20.543203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:20.543138 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:22.317323 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:22.317285 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:24.151790 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:24.151744 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:25.887598 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:25.887553 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:28.056848 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:28.056811 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:30.254007 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:30.253973 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:32.072737 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:32.072695 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:33.930819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:33.930778 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:34.737659 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:34.737600 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:35.803351 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:35.803317 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:37.256717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:37.256675 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:37.539812 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:37.539722 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:39.322021 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:39.321987 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:41.120519 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:41.120486 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:42.936745 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:42.936689 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:43.141591 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:33:43.141547 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:33:44.695779 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:44.695738 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:45.531046 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:45.531012 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:46.377694 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:46.377632 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:48.315512 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:48.315475 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:50.231438 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:50.231396 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:52.031079 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:52.031031 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:53.785052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:53.785016 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:55.740004 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:55.739965 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:57.592156 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:57.592120 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:58.716970 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:58.716933 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:33:59.473789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:33:59.473753 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:01.261725 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:01.261687 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:03.066450 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:03.066415 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:04.796603 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:04.796561 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:06.644031 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:06.643988 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:08.432522 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:08.432487 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:09.696635 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:09.696589 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:10.289642 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:10.289609 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:12.082563 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:12.082530 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:13.141948 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:34:13.141902 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:34:13.819469 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:13.819429 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:15.619952 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:15.619904 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:17.602615 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:17.602585 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:19.548953 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:19.548917 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:21.326859 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:21.326826 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:23.118140 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:23.118088 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:24.899435 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:24.899391 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:26.646152 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:26.646116 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:28.421543 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:28.421500 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:30.527418 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:30.527375 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:32.451707 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:32.451673 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:34.203481 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:34.203446 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:36.027828 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:36.027793 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:37.833511 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:37.833457 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:39.612897 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:39.612863 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:41.310769 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:41.310723 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:43.141761 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:34:43.141724 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:34:43.282533 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:43.282496 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:45.075599 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:45.075559 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:46.810748 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:46.810708 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:48.524225 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:48.524189 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:50.312274 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:50.312235 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:50.601555 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:50.601475 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:52.093603 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:52.093551 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:53.811100 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:53.811065 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:53.969594 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:53.969556 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr"] Nov 02 08:34:54.061152 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:54.061029 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:55.819486 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:55.819453 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:57.595746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:57.595712 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:34:59.316704 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:34:59.316664 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:01.097851 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:01.097813 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:02.686629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:02.686591 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:02.891180 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:02.891125 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:03.690020 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:03.689961 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:04.149387 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:04.149349 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:04.743928 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:04.743893 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:06.433844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:06.433811 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:08.176584 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:08.176556 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:09.002386 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.002339 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" podUID="345e41d9-05b3-4795-a9fc-fce423798f9d" containerName="main" containerID="cri-o://cbd930a5c62798d634223a6c7b9a073bf91a1435c521093824eadd273ab3f0d5" gracePeriod=45 Nov 02 08:35:09.263377 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.263342 5899 generic.go:356] "Generic (PLEG): container finished" podID="345e41d9-05b3-4795-a9fc-fce423798f9d" containerID="cbd930a5c62798d634223a6c7b9a073bf91a1435c521093824eadd273ab3f0d5" exitCode=0 Nov 02 08:35:09.263719 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.263428 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" event={"ID":"345e41d9-05b3-4795-a9fc-fce423798f9d","Type":"ContainerDied","Data":"cbd930a5c62798d634223a6c7b9a073bf91a1435c521093824eadd273ab3f0d5"} Nov 02 08:35:09.263719 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.263482 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" event={"ID":"345e41d9-05b3-4795-a9fc-fce423798f9d","Type":"ContainerDied","Data":"8b248fb8d39b1ca99bb065c6950f4c15a5a20e11cfd9ea334b85e7817154d8f6"} Nov 02 08:35:09.263719 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.263520 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b248fb8d39b1ca99bb065c6950f4c15a5a20e11cfd9ea334b85e7817154d8f6" Nov 02 08:35:09.291192 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.290986 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:35:09.480712 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.480676 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/345e41d9-05b3-4795-a9fc-fce423798f9d-kube-api-access-crndz\" (UniqueName: \"kubernetes.io/projected/345e41d9-05b3-4795-a9fc-fce423798f9d-kube-api-access-crndz\") pod \"345e41d9-05b3-4795-a9fc-fce423798f9d\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " Nov 02 08:35:09.480900 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.480754 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/345e41d9-05b3-4795-a9fc-fce423798f9d-tls-certs\" (UniqueName: \"kubernetes.io/secret/345e41d9-05b3-4795-a9fc-fce423798f9d-tls-certs\") pod \"345e41d9-05b3-4795-a9fc-fce423798f9d\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " Nov 02 08:35:09.480900 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.480798 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-home\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-home\") pod \"345e41d9-05b3-4795-a9fc-fce423798f9d\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " Nov 02 08:35:09.480900 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.480843 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-tmp-dir\") pod \"345e41d9-05b3-4795-a9fc-fce423798f9d\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " Nov 02 08:35:09.481073 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.480951 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-dshm\") pod \"345e41d9-05b3-4795-a9fc-fce423798f9d\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " Nov 02 08:35:09.481073 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.480989 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-model-cache\") pod \"345e41d9-05b3-4795-a9fc-fce423798f9d\" (UID: \"345e41d9-05b3-4795-a9fc-fce423798f9d\") " Nov 02 08:35:09.481243 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.481133 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-home" pod "345e41d9-05b3-4795-a9fc-fce423798f9d" (UID: "345e41d9-05b3-4795-a9fc-fce423798f9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:09.481403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.481373 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-model-cache" pod "345e41d9-05b3-4795-a9fc-fce423798f9d" (UID: "345e41d9-05b3-4795-a9fc-fce423798f9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:09.481553 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.481533 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:09.481615 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.481557 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:09.481615 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.481592 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-tmp-dir" pod "345e41d9-05b3-4795-a9fc-fce423798f9d" (UID: "345e41d9-05b3-4795-a9fc-fce423798f9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:09.483085 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.483057 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/345e41d9-05b3-4795-a9fc-fce423798f9d-tls-certs" pod "345e41d9-05b3-4795-a9fc-fce423798f9d" (UID: "345e41d9-05b3-4795-a9fc-fce423798f9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:35:09.483242 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.483218 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345e41d9-05b3-4795-a9fc-fce423798f9d-kube-api-access-crndz" pod "345e41d9-05b3-4795-a9fc-fce423798f9d" (UID: "345e41d9-05b3-4795-a9fc-fce423798f9d"). InnerVolumeSpecName "kube-api-access-crndz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:09.483310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.483263 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-dshm" pod "345e41d9-05b3-4795-a9fc-fce423798f9d" (UID: "345e41d9-05b3-4795-a9fc-fce423798f9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:09.582577 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.582492 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:09.582577 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.582524 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/345e41d9-05b3-4795-a9fc-fce423798f9d-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:09.582577 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.582535 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crndz\" (UniqueName: \"kubernetes.io/projected/345e41d9-05b3-4795-a9fc-fce423798f9d-kube-api-access-crndz\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:09.582577 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.582545 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/345e41d9-05b3-4795-a9fc-fce423798f9d-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:09.904439 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:09.904351 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:10.161138 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:10.161036 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:10.269804 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:10.269768 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr" Nov 02 08:35:10.324409 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:10.324376 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr"] Nov 02 08:35:10.337221 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:10.337192 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-12df3c/rate-test-model-12df3c-kserve-79d87cb76f-5zbrr"] Nov 02 08:35:10.861898 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:10.861858 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:10.997960 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:10.997928 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:11.821510 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:11.821476 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="345e41d9-05b3-4795-a9fc-fce423798f9d" path="/var/lib/kubelet/pods/345e41d9-05b3-4795-a9fc-fce423798f9d/volumes" Nov 02 08:35:12.027637 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:12.027591 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:12.239378 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:12.239349 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:13.141784 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:35:13.141747 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:35:13.807528 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:13.807486 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:15.212689 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.212651 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz"] Nov 02 08:35:15.213114 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.212908 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" podUID="8f09bad9-8a3f-45ba-93c3-3bc09e153761" containerName="maas-api" containerID="cri-o://4342aab66622b90378de9be23b1ce2bf1ddca7664c08f0b3c2bcd1d5efb41a6a" gracePeriod=30 Nov 02 08:35:15.310215 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.310176 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:15.352440 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.352395 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" podUID="8f09bad9-8a3f-45ba-93c3-3bc09e153761" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.109:8443/health\": dial tcp 10.134.0.109:8443: connect: connection refused" Nov 02 08:35:15.496467 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.496433 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 08:35:15.552413 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.552377 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-maas-api-tls\") pod \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " Nov 02 08:35:15.552592 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.552532 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-metrics-certs\") pod \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " Nov 02 08:35:15.552674 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.552609 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f09bad9-8a3f-45ba-93c3-3bc09e153761-kube-api-access-kgb55\" (UniqueName: \"kubernetes.io/projected/8f09bad9-8a3f-45ba-93c3-3bc09e153761-kube-api-access-kgb55\") pod \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " Nov 02 08:35:15.552674 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.552645 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f09bad9-8a3f-45ba-93c3-3bc09e153761-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f09bad9-8a3f-45ba-93c3-3bc09e153761-openshift-service-ca\") pod \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\" (UID: \"8f09bad9-8a3f-45ba-93c3-3bc09e153761\") " Nov 02 08:35:15.553070 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.553041 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f09bad9-8a3f-45ba-93c3-3bc09e153761-openshift-service-ca" pod "8f09bad9-8a3f-45ba-93c3-3bc09e153761" (UID: "8f09bad9-8a3f-45ba-93c3-3bc09e153761"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:35:15.556203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.554828 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-metrics-certs" pod "8f09bad9-8a3f-45ba-93c3-3bc09e153761" (UID: "8f09bad9-8a3f-45ba-93c3-3bc09e153761"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:35:15.556203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.554925 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-maas-api-tls" pod "8f09bad9-8a3f-45ba-93c3-3bc09e153761" (UID: "8f09bad9-8a3f-45ba-93c3-3bc09e153761"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:35:15.556203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.555507 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f09bad9-8a3f-45ba-93c3-3bc09e153761-kube-api-access-kgb55" pod "8f09bad9-8a3f-45ba-93c3-3bc09e153761" (UID: "8f09bad9-8a3f-45ba-93c3-3bc09e153761"). InnerVolumeSpecName "kube-api-access-kgb55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:15.654361 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.654325 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:15.654361 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.654354 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f09bad9-8a3f-45ba-93c3-3bc09e153761-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:15.654361 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.654366 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgb55\" (UniqueName: \"kubernetes.io/projected/8f09bad9-8a3f-45ba-93c3-3bc09e153761-kube-api-access-kgb55\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:15.654587 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.654375 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f09bad9-8a3f-45ba-93c3-3bc09e153761-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:15.920712 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:15.920676 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:16.305892 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:16.305858 5899 generic.go:356] "Generic (PLEG): container finished" podID="8f09bad9-8a3f-45ba-93c3-3bc09e153761" containerID="4342aab66622b90378de9be23b1ce2bf1ddca7664c08f0b3c2bcd1d5efb41a6a" exitCode=0 Nov 02 08:35:16.306319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:16.305906 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" Nov 02 08:35:16.306319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:16.305944 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" event={"ID":"8f09bad9-8a3f-45ba-93c3-3bc09e153761","Type":"ContainerDied","Data":"4342aab66622b90378de9be23b1ce2bf1ddca7664c08f0b3c2bcd1d5efb41a6a"} Nov 02 08:35:16.306319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:16.305997 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz" event={"ID":"8f09bad9-8a3f-45ba-93c3-3bc09e153761","Type":"ContainerDied","Data":"565ca3aeac7e1cd4d739c4176fd7084f71e74491ed22d63681f5355f475ad098"} Nov 02 08:35:16.306319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:16.306020 5899 scope.go:122] "RemoveContainer" containerID="4342aab66622b90378de9be23b1ce2bf1ddca7664c08f0b3c2bcd1d5efb41a6a" Nov 02 08:35:16.315772 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:16.315757 5899 scope.go:122] "RemoveContainer" containerID="4342aab66622b90378de9be23b1ce2bf1ddca7664c08f0b3c2bcd1d5efb41a6a" Nov 02 08:35:16.316018 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:35:16.315994 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4342aab66622b90378de9be23b1ce2bf1ddca7664c08f0b3c2bcd1d5efb41a6a\": container with ID starting with 4342aab66622b90378de9be23b1ce2bf1ddca7664c08f0b3c2bcd1d5efb41a6a not found: ID does not exist" containerID="4342aab66622b90378de9be23b1ce2bf1ddca7664c08f0b3c2bcd1d5efb41a6a" Nov 02 08:35:16.316147 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:16.316024 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4342aab66622b90378de9be23b1ce2bf1ddca7664c08f0b3c2bcd1d5efb41a6a"} err="failed to get container status \"4342aab66622b90378de9be23b1ce2bf1ddca7664c08f0b3c2bcd1d5efb41a6a\": rpc error: code = NotFound desc = could not find container \"4342aab66622b90378de9be23b1ce2bf1ddca7664c08f0b3c2bcd1d5efb41a6a\": container with ID starting with 4342aab66622b90378de9be23b1ce2bf1ddca7664c08f0b3c2bcd1d5efb41a6a not found: ID does not exist" Nov 02 08:35:16.360906 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:16.360877 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz"] Nov 02 08:35:16.373745 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:16.373717 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-696c7a-584b59866c-gjdrz"] Nov 02 08:35:17.666475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:17.666441 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:17.820292 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:17.820256 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f09bad9-8a3f-45ba-93c3-3bc09e153761" path="/var/lib/kubelet/pods/8f09bad9-8a3f-45ba-93c3-3bc09e153761/volumes" Nov 02 08:35:17.852038 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:17.852007 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:18.157342 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:18.157303 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:18.657349 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:18.657296 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:19.149868 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:19.149831 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:19.809492 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:19.809457 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:19.952232 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:19.952140 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:21.137400 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:21.137360 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775"] Nov 02 08:35:21.219217 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:21.219143 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:21.604100 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:21.604062 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:23.307133 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:23.307092 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:25.202543 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:25.202498 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:25.737550 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:25.737510 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:25.884397 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:25.884365 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:26.404011 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:26.403981 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:26.621292 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:26.621255 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg"] Nov 02 08:35:26.653015 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:26.652980 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l"] Nov 02 08:35:26.791438 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:26.791409 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:26.992128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:26.992087 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:27.133834 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:27.133740 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:28.786599 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:28.786566 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:30.513213 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:30.513179 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:31.293535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:31.293499 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:31.622486 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:31.622385 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" podUID="0915e267-30ab-478c-a749-0fb605399ee4" containerName="payload-processing" containerID="cri-o://bbaaaea491b9ea6d401bc3e7cce6b94fb9ac479b3957a1f72e452b5d21f1dae9" gracePeriod=25 Nov 02 08:35:31.654174 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:31.654095 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" podUID="7bda83a4-b696-45c0-b3a2-9b03e75fe3d6" containerName="payload-pre-processing" containerID="cri-o://e00623cdfbdf61f9ee5c311b7cfe483515db4baa83beba6d794000ce8ec8be7a" gracePeriod=25 Nov 02 08:35:31.861702 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:31.861664 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:32.540137 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:32.540100 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:32.978874 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:32.978843 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:34.406548 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:34.406513 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:36.131532 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.131496 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:36.158453 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.158407 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" podUID="053fdd58-57a4-42a5-9cd6-4891d05da186" containerName="main" containerID="cri-o://d3bee85d591775d0b4698ab0e2a1a78f48486ee09b416a5ecb01827365702dbe" gracePeriod=45 Nov 02 08:35:36.425535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.425506 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:35:36.428829 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.428798 5899 generic.go:356] "Generic (PLEG): container finished" podID="053fdd58-57a4-42a5-9cd6-4891d05da186" containerID="d3bee85d591775d0b4698ab0e2a1a78f48486ee09b416a5ecb01827365702dbe" exitCode=0 Nov 02 08:35:36.428994 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.428837 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" Nov 02 08:35:36.428994 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.428877 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" event={"ID":"053fdd58-57a4-42a5-9cd6-4891d05da186","Type":"ContainerDied","Data":"d3bee85d591775d0b4698ab0e2a1a78f48486ee09b416a5ecb01827365702dbe"} Nov 02 08:35:36.428994 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.428923 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775" event={"ID":"053fdd58-57a4-42a5-9cd6-4891d05da186","Type":"ContainerDied","Data":"6a636244950a4211bf30d5baee67cc350f0e163526ef27e060ca4612a1b7d5db"} Nov 02 08:35:36.428994 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.428941 5899 scope.go:122] "RemoveContainer" containerID="d3bee85d591775d0b4698ab0e2a1a78f48486ee09b416a5ecb01827365702dbe" Nov 02 08:35:36.438585 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.438559 5899 scope.go:122] "RemoveContainer" containerID="d3bee85d591775d0b4698ab0e2a1a78f48486ee09b416a5ecb01827365702dbe" Nov 02 08:35:36.438908 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:35:36.438879 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3bee85d591775d0b4698ab0e2a1a78f48486ee09b416a5ecb01827365702dbe\": container with ID starting with d3bee85d591775d0b4698ab0e2a1a78f48486ee09b416a5ecb01827365702dbe not found: ID does not exist" containerID="d3bee85d591775d0b4698ab0e2a1a78f48486ee09b416a5ecb01827365702dbe" Nov 02 08:35:36.439016 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.438916 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3bee85d591775d0b4698ab0e2a1a78f48486ee09b416a5ecb01827365702dbe"} err="failed to get container status \"d3bee85d591775d0b4698ab0e2a1a78f48486ee09b416a5ecb01827365702dbe\": rpc error: code = NotFound desc = could not find container \"d3bee85d591775d0b4698ab0e2a1a78f48486ee09b416a5ecb01827365702dbe\": container with ID starting with d3bee85d591775d0b4698ab0e2a1a78f48486ee09b416a5ecb01827365702dbe not found: ID does not exist" Nov 02 08:35:36.600601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.600561 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/053fdd58-57a4-42a5-9cd6-4891d05da186-kube-api-access-gwz9z\" (UniqueName: \"kubernetes.io/projected/053fdd58-57a4-42a5-9cd6-4891d05da186-kube-api-access-gwz9z\") pod \"053fdd58-57a4-42a5-9cd6-4891d05da186\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " Nov 02 08:35:36.600800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.600608 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-model-cache\") pod \"053fdd58-57a4-42a5-9cd6-4891d05da186\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " Nov 02 08:35:36.600800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.600733 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-dshm\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-dshm\") pod \"053fdd58-57a4-42a5-9cd6-4891d05da186\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " Nov 02 08:35:36.600949 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.600873 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/053fdd58-57a4-42a5-9cd6-4891d05da186-tls-certs\" (UniqueName: \"kubernetes.io/secret/053fdd58-57a4-42a5-9cd6-4891d05da186-tls-certs\") pod \"053fdd58-57a4-42a5-9cd6-4891d05da186\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " Nov 02 08:35:36.600949 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.600909 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-home\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-home\") pod \"053fdd58-57a4-42a5-9cd6-4891d05da186\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " Nov 02 08:35:36.601068 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.600949 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-model-cache" pod "053fdd58-57a4-42a5-9cd6-4891d05da186" (UID: "053fdd58-57a4-42a5-9cd6-4891d05da186"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:36.601068 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.600960 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-tmp-dir\") pod \"053fdd58-57a4-42a5-9cd6-4891d05da186\" (UID: \"053fdd58-57a4-42a5-9cd6-4891d05da186\") " Nov 02 08:35:36.601255 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.601234 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-home" pod "053fdd58-57a4-42a5-9cd6-4891d05da186" (UID: "053fdd58-57a4-42a5-9cd6-4891d05da186"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:36.601337 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.601270 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-tmp-dir" pod "053fdd58-57a4-42a5-9cd6-4891d05da186" (UID: "053fdd58-57a4-42a5-9cd6-4891d05da186"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:36.601533 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.601513 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:36.601598 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.601564 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:36.601598 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.601584 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:36.602982 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.602957 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-dshm" pod "053fdd58-57a4-42a5-9cd6-4891d05da186" (UID: "053fdd58-57a4-42a5-9cd6-4891d05da186"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:35:36.603091 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.603006 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053fdd58-57a4-42a5-9cd6-4891d05da186-kube-api-access-gwz9z" pod "053fdd58-57a4-42a5-9cd6-4891d05da186" (UID: "053fdd58-57a4-42a5-9cd6-4891d05da186"). InnerVolumeSpecName "kube-api-access-gwz9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:36.603091 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.603009 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/053fdd58-57a4-42a5-9cd6-4891d05da186-tls-certs" pod "053fdd58-57a4-42a5-9cd6-4891d05da186" (UID: "053fdd58-57a4-42a5-9cd6-4891d05da186"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:35:36.702751 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.702661 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/053fdd58-57a4-42a5-9cd6-4891d05da186-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:36.702751 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.702698 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/053fdd58-57a4-42a5-9cd6-4891d05da186-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:36.702751 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.702711 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwz9z\" (UniqueName: \"kubernetes.io/projected/053fdd58-57a4-42a5-9cd6-4891d05da186-kube-api-access-gwz9z\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:36.778047 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.777967 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775"] Nov 02 08:35:36.789716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:36.789684 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9877ce/sub-test-model-9877ce-kserve-649776c967-h6775"] Nov 02 08:35:37.357019 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:37.356973 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:37.491673 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:37.491633 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:37.770561 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:37.770520 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" podUID="7bda83a4-b696-45c0-b3a2-9b03e75fe3d6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:35:37.770561 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:37.770548 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" podUID="0915e267-30ab-478c-a749-0fb605399ee4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:35:37.836508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:37.836473 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="053fdd58-57a4-42a5-9cd6-4891d05da186" path="/var/lib/kubelet/pods/053fdd58-57a4-42a5-9cd6-4891d05da186/volumes" Nov 02 08:35:37.986033 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:37.985998 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:38.817986 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:38.817942 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:39.602286 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:39.602226 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:41.342700 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:41.342611 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:43.141861 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:35:43.141821 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:35:43.205172 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:43.205122 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:44.849401 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:44.849366 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:44.997876 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:44.997830 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:46.710594 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:46.710562 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:47.770137 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:47.770092 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" podUID="7bda83a4-b696-45c0-b3a2-9b03e75fe3d6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:35:47.770561 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:47.770147 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" podUID="0915e267-30ab-478c-a749-0fb605399ee4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:35:48.143567 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.143470 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f"] Nov 02 08:35:48.143784 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.143752 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" podUID="36f9a988-8ac6-4969-8e0e-f7a267376b20" containerName="maas-api" containerID="cri-o://f9565de779d20ca015ac863d74447b97c3de64c2888db5ad983b41b81c719604" gracePeriod=30 Nov 02 08:35:48.252718 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.252673 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:48.444514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.444482 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:35:48.505762 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.505723 5899 generic.go:356] "Generic (PLEG): container finished" podID="36f9a988-8ac6-4969-8e0e-f7a267376b20" containerID="f9565de779d20ca015ac863d74447b97c3de64c2888db5ad983b41b81c719604" exitCode=0 Nov 02 08:35:48.505762 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.505764 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" Nov 02 08:35:48.505991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.505813 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" event={"ID":"36f9a988-8ac6-4969-8e0e-f7a267376b20","Type":"ContainerDied","Data":"f9565de779d20ca015ac863d74447b97c3de64c2888db5ad983b41b81c719604"} Nov 02 08:35:48.505991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.505883 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f" event={"ID":"36f9a988-8ac6-4969-8e0e-f7a267376b20","Type":"ContainerDied","Data":"61081da5d937e2900e9125a2e24a290a526fe105c3f8849d37307a5062c0a52b"} Nov 02 08:35:48.505991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.505921 5899 scope.go:122] "RemoveContainer" containerID="f9565de779d20ca015ac863d74447b97c3de64c2888db5ad983b41b81c719604" Nov 02 08:35:48.518247 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.518226 5899 scope.go:122] "RemoveContainer" containerID="f9565de779d20ca015ac863d74447b97c3de64c2888db5ad983b41b81c719604" Nov 02 08:35:48.518552 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:35:48.518531 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9565de779d20ca015ac863d74447b97c3de64c2888db5ad983b41b81c719604\": container with ID starting with f9565de779d20ca015ac863d74447b97c3de64c2888db5ad983b41b81c719604 not found: ID does not exist" containerID="f9565de779d20ca015ac863d74447b97c3de64c2888db5ad983b41b81c719604" Nov 02 08:35:48.518603 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.518561 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9565de779d20ca015ac863d74447b97c3de64c2888db5ad983b41b81c719604"} err="failed to get container status \"f9565de779d20ca015ac863d74447b97c3de64c2888db5ad983b41b81c719604\": rpc error: code = NotFound desc = could not find container \"f9565de779d20ca015ac863d74447b97c3de64c2888db5ad983b41b81c719604\": container with ID starting with f9565de779d20ca015ac863d74447b97c3de64c2888db5ad983b41b81c719604 not found: ID does not exist" Nov 02 08:35:48.538643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.538615 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f9a988-8ac6-4969-8e0e-f7a267376b20-kube-api-access-mhjsg\" (UniqueName: \"kubernetes.io/projected/36f9a988-8ac6-4969-8e0e-f7a267376b20-kube-api-access-mhjsg\") pod \"36f9a988-8ac6-4969-8e0e-f7a267376b20\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " Nov 02 08:35:48.538840 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.538690 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-maas-api-tls\") pod \"36f9a988-8ac6-4969-8e0e-f7a267376b20\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " Nov 02 08:35:48.538840 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.538822 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-metrics-certs\") pod \"36f9a988-8ac6-4969-8e0e-f7a267376b20\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " Nov 02 08:35:48.538948 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.538858 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f9a988-8ac6-4969-8e0e-f7a267376b20-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f9a988-8ac6-4969-8e0e-f7a267376b20-openshift-service-ca\") pod \"36f9a988-8ac6-4969-8e0e-f7a267376b20\" (UID: \"36f9a988-8ac6-4969-8e0e-f7a267376b20\") " Nov 02 08:35:48.539425 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.539400 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f9a988-8ac6-4969-8e0e-f7a267376b20-openshift-service-ca" pod "36f9a988-8ac6-4969-8e0e-f7a267376b20" (UID: "36f9a988-8ac6-4969-8e0e-f7a267376b20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:35:48.540844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.540821 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f9a988-8ac6-4969-8e0e-f7a267376b20-kube-api-access-mhjsg" pod "36f9a988-8ac6-4969-8e0e-f7a267376b20" (UID: "36f9a988-8ac6-4969-8e0e-f7a267376b20"). InnerVolumeSpecName "kube-api-access-mhjsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:48.540935 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.540903 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-maas-api-tls" pod "36f9a988-8ac6-4969-8e0e-f7a267376b20" (UID: "36f9a988-8ac6-4969-8e0e-f7a267376b20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:35:48.540935 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.540919 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-metrics-certs" pod "36f9a988-8ac6-4969-8e0e-f7a267376b20" (UID: "36f9a988-8ac6-4969-8e0e-f7a267376b20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:35:48.642223 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.641022 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhjsg\" (UniqueName: \"kubernetes.io/projected/36f9a988-8ac6-4969-8e0e-f7a267376b20-kube-api-access-mhjsg\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:48.642223 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.641062 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:48.642223 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.641076 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f9a988-8ac6-4969-8e0e-f7a267376b20-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:48.642223 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.641096 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f9a988-8ac6-4969-8e0e-f7a267376b20-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:48.853137 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.853097 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f"] Nov 02 08:35:48.857224 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.857192 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-643355-67f4bffd44-5rj9f"] Nov 02 08:35:48.867091 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:48.867063 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:49.819924 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:49.819889 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f9a988-8ac6-4969-8e0e-f7a267376b20" path="/var/lib/kubelet/pods/36f9a988-8ac6-4969-8e0e-f7a267376b20/volumes" Nov 02 08:35:50.562527 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:50.562472 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:51.902414 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:51.902381 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m"] Nov 02 08:35:51.902814 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:51.902645 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" podUID="73ee734d-db5b-4d7f-aff3-044bb4802ba1" containerName="maas-api" containerID="cri-o://c39e7d1872f13925c5c7dd2e5abff161445f32dc93b2bdc8a62cf590274e9bc4" gracePeriod=30 Nov 02 08:35:51.988793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:51.988763 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" Nov 02 08:35:52.016671 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.016636 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:52.017394 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.017365 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" Nov 02 08:35:52.076742 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.076711 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0915e267-30ab-478c-a749-0fb605399ee4-kube-api-access-vp7qn\" (UniqueName: \"kubernetes.io/projected/0915e267-30ab-478c-a749-0fb605399ee4-kube-api-access-vp7qn\") pod \"0915e267-30ab-478c-a749-0fb605399ee4\" (UID: \"0915e267-30ab-478c-a749-0fb605399ee4\") " Nov 02 08:35:52.076942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.076766 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0915e267-30ab-478c-a749-0fb605399ee4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0915e267-30ab-478c-a749-0fb605399ee4-plugins-config-volume\") pod \"0915e267-30ab-478c-a749-0fb605399ee4\" (UID: \"0915e267-30ab-478c-a749-0fb605399ee4\") " Nov 02 08:35:52.076942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.076843 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-kube-api-access-6fwk8\" (UniqueName: \"kubernetes.io/projected/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-kube-api-access-6fwk8\") pod \"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6\" (UID: \"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6\") " Nov 02 08:35:52.077066 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.076951 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-plugins-config-volume\") pod \"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6\" (UID: \"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6\") " Nov 02 08:35:52.077902 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.077566 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0915e267-30ab-478c-a749-0fb605399ee4-plugins-config-volume" pod "0915e267-30ab-478c-a749-0fb605399ee4" (UID: "0915e267-30ab-478c-a749-0fb605399ee4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:35:52.077902 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.077659 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-plugins-config-volume" pod "7bda83a4-b696-45c0-b3a2-9b03e75fe3d6" (UID: "7bda83a4-b696-45c0-b3a2-9b03e75fe3d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:35:52.079864 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.079838 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0915e267-30ab-478c-a749-0fb605399ee4-kube-api-access-vp7qn" pod "0915e267-30ab-478c-a749-0fb605399ee4" (UID: "0915e267-30ab-478c-a749-0fb605399ee4"). InnerVolumeSpecName "kube-api-access-vp7qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:52.080092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.080068 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-kube-api-access-6fwk8" pod "7bda83a4-b696-45c0-b3a2-9b03e75fe3d6" (UID: "7bda83a4-b696-45c0-b3a2-9b03e75fe3d6"). InnerVolumeSpecName "kube-api-access-6fwk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:52.178529 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.178453 5899 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-plugins-config-volume\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:52.178529 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.178490 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp7qn\" (UniqueName: \"kubernetes.io/projected/0915e267-30ab-478c-a749-0fb605399ee4-kube-api-access-vp7qn\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:52.178529 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.178515 5899 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0915e267-30ab-478c-a749-0fb605399ee4-plugins-config-volume\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:52.178529 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.178530 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fwk8\" (UniqueName: \"kubernetes.io/projected/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6-kube-api-access-6fwk8\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:52.190819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.190784 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 08:35:52.279921 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.279887 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ee734d-db5b-4d7f-aff3-044bb4802ba1-kube-api-access-kgzsr\" (UniqueName: \"kubernetes.io/projected/73ee734d-db5b-4d7f-aff3-044bb4802ba1-kube-api-access-kgzsr\") pod \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " Nov 02 08:35:52.280103 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.279959 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-maas-api-tls\") pod \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " Nov 02 08:35:52.280103 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.280048 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ee734d-db5b-4d7f-aff3-044bb4802ba1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ee734d-db5b-4d7f-aff3-044bb4802ba1-openshift-service-ca\") pod \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " Nov 02 08:35:52.280247 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.280230 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-metrics-certs\") pod \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\" (UID: \"73ee734d-db5b-4d7f-aff3-044bb4802ba1\") " Nov 02 08:35:52.280464 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.280434 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ee734d-db5b-4d7f-aff3-044bb4802ba1-openshift-service-ca" pod "73ee734d-db5b-4d7f-aff3-044bb4802ba1" (UID: "73ee734d-db5b-4d7f-aff3-044bb4802ba1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:35:52.280692 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.280674 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ee734d-db5b-4d7f-aff3-044bb4802ba1-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:52.282278 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.282254 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-metrics-certs" pod "73ee734d-db5b-4d7f-aff3-044bb4802ba1" (UID: "73ee734d-db5b-4d7f-aff3-044bb4802ba1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:35:52.282377 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.282275 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-maas-api-tls" pod "73ee734d-db5b-4d7f-aff3-044bb4802ba1" (UID: "73ee734d-db5b-4d7f-aff3-044bb4802ba1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:35:52.282377 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.282313 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ee734d-db5b-4d7f-aff3-044bb4802ba1-kube-api-access-kgzsr" pod "73ee734d-db5b-4d7f-aff3-044bb4802ba1" (UID: "73ee734d-db5b-4d7f-aff3-044bb4802ba1"). InnerVolumeSpecName "kube-api-access-kgzsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:35:52.382660 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.382620 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:52.382660 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.382662 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgzsr\" (UniqueName: \"kubernetes.io/projected/73ee734d-db5b-4d7f-aff3-044bb4802ba1-kube-api-access-kgzsr\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:52.382914 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.382676 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ee734d-db5b-4d7f-aff3-044bb4802ba1-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:35:52.531951 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.531909 5899 generic.go:356] "Generic (PLEG): container finished" podID="7bda83a4-b696-45c0-b3a2-9b03e75fe3d6" containerID="e00623cdfbdf61f9ee5c311b7cfe483515db4baa83beba6d794000ce8ec8be7a" exitCode=0 Nov 02 08:35:52.532185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.531965 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" Nov 02 08:35:52.532185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.531991 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" event={"ID":"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6","Type":"ContainerDied","Data":"e00623cdfbdf61f9ee5c311b7cfe483515db4baa83beba6d794000ce8ec8be7a"} Nov 02 08:35:52.532185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.532049 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l" event={"ID":"7bda83a4-b696-45c0-b3a2-9b03e75fe3d6","Type":"ContainerDied","Data":"6285ff1378a45c4966ed6783e392230232284ec7a2efcdf60ac740a96ca12df4"} Nov 02 08:35:52.532185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.532073 5899 scope.go:122] "RemoveContainer" containerID="e00623cdfbdf61f9ee5c311b7cfe483515db4baa83beba6d794000ce8ec8be7a" Nov 02 08:35:52.533538 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.533515 5899 generic.go:356] "Generic (PLEG): container finished" podID="73ee734d-db5b-4d7f-aff3-044bb4802ba1" containerID="c39e7d1872f13925c5c7dd2e5abff161445f32dc93b2bdc8a62cf590274e9bc4" exitCode=0 Nov 02 08:35:52.533668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.533569 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" Nov 02 08:35:52.533668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.533601 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" event={"ID":"73ee734d-db5b-4d7f-aff3-044bb4802ba1","Type":"ContainerDied","Data":"c39e7d1872f13925c5c7dd2e5abff161445f32dc93b2bdc8a62cf590274e9bc4"} Nov 02 08:35:52.533784 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.533665 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m" event={"ID":"73ee734d-db5b-4d7f-aff3-044bb4802ba1","Type":"ContainerDied","Data":"ea93cfa0d793d045107b0fdba4cc31e4503544c5ab585629780b044b14193ea9"} Nov 02 08:35:52.535113 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.535011 5899 generic.go:356] "Generic (PLEG): container finished" podID="0915e267-30ab-478c-a749-0fb605399ee4" containerID="bbaaaea491b9ea6d401bc3e7cce6b94fb9ac479b3957a1f72e452b5d21f1dae9" exitCode=0 Nov 02 08:35:52.535113 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.535055 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" event={"ID":"0915e267-30ab-478c-a749-0fb605399ee4","Type":"ContainerDied","Data":"bbaaaea491b9ea6d401bc3e7cce6b94fb9ac479b3957a1f72e452b5d21f1dae9"} Nov 02 08:35:52.535113 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.535063 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" Nov 02 08:35:52.535113 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.535085 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg" event={"ID":"0915e267-30ab-478c-a749-0fb605399ee4","Type":"ContainerDied","Data":"095f5c3ca076fca10072fe8cf2fce29c1c862779f03cb7fde2d1b4629da84ac5"} Nov 02 08:35:52.542930 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.542908 5899 scope.go:122] "RemoveContainer" containerID="e00623cdfbdf61f9ee5c311b7cfe483515db4baa83beba6d794000ce8ec8be7a" Nov 02 08:35:52.543222 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:35:52.543198 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00623cdfbdf61f9ee5c311b7cfe483515db4baa83beba6d794000ce8ec8be7a\": container with ID starting with e00623cdfbdf61f9ee5c311b7cfe483515db4baa83beba6d794000ce8ec8be7a not found: ID does not exist" containerID="e00623cdfbdf61f9ee5c311b7cfe483515db4baa83beba6d794000ce8ec8be7a" Nov 02 08:35:52.543297 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.543227 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00623cdfbdf61f9ee5c311b7cfe483515db4baa83beba6d794000ce8ec8be7a"} err="failed to get container status \"e00623cdfbdf61f9ee5c311b7cfe483515db4baa83beba6d794000ce8ec8be7a\": rpc error: code = NotFound desc = could not find container \"e00623cdfbdf61f9ee5c311b7cfe483515db4baa83beba6d794000ce8ec8be7a\": container with ID starting with e00623cdfbdf61f9ee5c311b7cfe483515db4baa83beba6d794000ce8ec8be7a not found: ID does not exist" Nov 02 08:35:52.543297 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.543246 5899 scope.go:122] "RemoveContainer" containerID="c39e7d1872f13925c5c7dd2e5abff161445f32dc93b2bdc8a62cf590274e9bc4" Nov 02 08:35:52.552994 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.552973 5899 scope.go:122] "RemoveContainer" containerID="c39e7d1872f13925c5c7dd2e5abff161445f32dc93b2bdc8a62cf590274e9bc4" Nov 02 08:35:52.553394 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:35:52.553374 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39e7d1872f13925c5c7dd2e5abff161445f32dc93b2bdc8a62cf590274e9bc4\": container with ID starting with c39e7d1872f13925c5c7dd2e5abff161445f32dc93b2bdc8a62cf590274e9bc4 not found: ID does not exist" containerID="c39e7d1872f13925c5c7dd2e5abff161445f32dc93b2bdc8a62cf590274e9bc4" Nov 02 08:35:52.553532 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.553403 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39e7d1872f13925c5c7dd2e5abff161445f32dc93b2bdc8a62cf590274e9bc4"} err="failed to get container status \"c39e7d1872f13925c5c7dd2e5abff161445f32dc93b2bdc8a62cf590274e9bc4\": rpc error: code = NotFound desc = could not find container \"c39e7d1872f13925c5c7dd2e5abff161445f32dc93b2bdc8a62cf590274e9bc4\": container with ID starting with c39e7d1872f13925c5c7dd2e5abff161445f32dc93b2bdc8a62cf590274e9bc4 not found: ID does not exist" Nov 02 08:35:52.553532 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.553418 5899 scope.go:122] "RemoveContainer" containerID="bbaaaea491b9ea6d401bc3e7cce6b94fb9ac479b3957a1f72e452b5d21f1dae9" Nov 02 08:35:52.562231 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.562213 5899 scope.go:122] "RemoveContainer" containerID="bbaaaea491b9ea6d401bc3e7cce6b94fb9ac479b3957a1f72e452b5d21f1dae9" Nov 02 08:35:52.562513 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:35:52.562493 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbaaaea491b9ea6d401bc3e7cce6b94fb9ac479b3957a1f72e452b5d21f1dae9\": container with ID starting with bbaaaea491b9ea6d401bc3e7cce6b94fb9ac479b3957a1f72e452b5d21f1dae9 not found: ID does not exist" containerID="bbaaaea491b9ea6d401bc3e7cce6b94fb9ac479b3957a1f72e452b5d21f1dae9" Nov 02 08:35:52.562562 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.562522 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbaaaea491b9ea6d401bc3e7cce6b94fb9ac479b3957a1f72e452b5d21f1dae9"} err="failed to get container status \"bbaaaea491b9ea6d401bc3e7cce6b94fb9ac479b3957a1f72e452b5d21f1dae9\": rpc error: code = NotFound desc = could not find container \"bbaaaea491b9ea6d401bc3e7cce6b94fb9ac479b3957a1f72e452b5d21f1dae9\": container with ID starting with bbaaaea491b9ea6d401bc3e7cce6b94fb9ac479b3957a1f72e452b5d21f1dae9 not found: ID does not exist" Nov 02 08:35:52.591478 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.591436 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m"] Nov 02 08:35:52.597540 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.597509 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c712b1-5864bf45c-5q56m"] Nov 02 08:35:52.649845 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.649815 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l"] Nov 02 08:35:52.654821 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.654789 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-643355-77d866896c-wwn2l"] Nov 02 08:35:52.691244 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.691209 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg"] Nov 02 08:35:52.693280 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.693258 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-643355-75b78974f7-6t2rg"] Nov 02 08:35:52.749925 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:52.749888 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:53.819380 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:53.819347 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0915e267-30ab-478c-a749-0fb605399ee4" path="/var/lib/kubelet/pods/0915e267-30ab-478c-a749-0fb605399ee4/volumes" Nov 02 08:35:53.819846 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:53.819830 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ee734d-db5b-4d7f-aff3-044bb4802ba1" path="/var/lib/kubelet/pods/73ee734d-db5b-4d7f-aff3-044bb4802ba1/volumes" Nov 02 08:35:53.820245 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:53.820231 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bda83a4-b696-45c0-b3a2-9b03e75fe3d6" path="/var/lib/kubelet/pods/7bda83a4-b696-45c0-b3a2-9b03e75fe3d6/volumes" Nov 02 08:35:54.497858 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:54.497817 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:56.160991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:56.160954 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:57.586123 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:57.586084 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:57.721627 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:57.721594 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:57.860142 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:57.860055 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:58.201947 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:58.201867 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:58.574822 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:58.574782 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:59.347982 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:59.347948 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:35:59.637412 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:35:59.637309 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:00.900092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:00.900058 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:01.036491 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:01.036458 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:01.577483 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:01.577448 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:01.760876 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:01.760838 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:02.156497 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:02.156463 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:02.951445 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:02.951408 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:03.437124 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:03.437083 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:05.102177 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:05.102128 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:05.736780 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:05.736741 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:05.878933 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:05.878900 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:06.225525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:06.225436 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:06.989513 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:06.989470 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:08.573091 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:08.573051 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz"] Nov 02 08:36:08.826442 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:08.826362 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:08.987511 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:08.987472 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:09.620816 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:09.620774 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:10.022150 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:10.022115 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:10.195710 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:10.195675 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:10.695690 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:10.695646 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:12.290499 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:12.290463 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:12.434377 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:12.434344 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:13.140601 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:36:13.140561 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:36:13.573653 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:13.573600 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" podUID="b660cf56-4367-4815-ae30-acbca8e0ce0b" containerName="payload-pre-processing" containerID="cri-o://3c127c305c2e7345b79252321ace8bc40d53b808a5767738717041dbcba28951" gracePeriod=25 Nov 02 08:36:14.017700 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:14.017661 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:14.911149 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:14.911110 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:15.044061 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:15.044020 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:15.467604 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:15.467569 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:15.700055 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:15.700018 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:17.526046 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:17.526007 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:19.468132 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:19.468088 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:20.649050 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:20.649003 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" podUID="b660cf56-4367-4815-ae30-acbca8e0ce0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:36:21.172028 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:21.171986 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:22.509674 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.509633 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt"] Nov 02 08:36:22.510075 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.509930 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" podUID="6d668935-89fc-4909-a4a8-8d0f476490d0" containerName="maas-api" containerID="cri-o://e0820b5150c23a7804a82f837f50806e8f9875010e01bbeeb88af3d8b648ebd9" gracePeriod=30 Nov 02 08:36:22.601211 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.601145 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:22.723282 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.723241 5899 generic.go:356] "Generic (PLEG): container finished" podID="6d668935-89fc-4909-a4a8-8d0f476490d0" containerID="e0820b5150c23a7804a82f837f50806e8f9875010e01bbeeb88af3d8b648ebd9" exitCode=0 Nov 02 08:36:22.723491 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.723354 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" event={"ID":"6d668935-89fc-4909-a4a8-8d0f476490d0","Type":"ContainerDied","Data":"e0820b5150c23a7804a82f837f50806e8f9875010e01bbeeb88af3d8b648ebd9"} Nov 02 08:36:22.819730 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.819701 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:36:22.961118 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.961067 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d668935-89fc-4909-a4a8-8d0f476490d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d668935-89fc-4909-a4a8-8d0f476490d0-openshift-service-ca\") pod \"6d668935-89fc-4909-a4a8-8d0f476490d0\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " Nov 02 08:36:22.961340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.961144 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-metrics-certs\") pod \"6d668935-89fc-4909-a4a8-8d0f476490d0\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " Nov 02 08:36:22.961340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.961240 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d668935-89fc-4909-a4a8-8d0f476490d0-kube-api-access-zsl9s\" (UniqueName: \"kubernetes.io/projected/6d668935-89fc-4909-a4a8-8d0f476490d0-kube-api-access-zsl9s\") pod \"6d668935-89fc-4909-a4a8-8d0f476490d0\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " Nov 02 08:36:22.961479 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.961348 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-maas-api-tls\") pod \"6d668935-89fc-4909-a4a8-8d0f476490d0\" (UID: \"6d668935-89fc-4909-a4a8-8d0f476490d0\") " Nov 02 08:36:22.961566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.961538 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d668935-89fc-4909-a4a8-8d0f476490d0-openshift-service-ca" pod "6d668935-89fc-4909-a4a8-8d0f476490d0" (UID: "6d668935-89fc-4909-a4a8-8d0f476490d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:36:22.962305 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.962278 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d668935-89fc-4909-a4a8-8d0f476490d0-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:36:22.963968 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.963934 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d668935-89fc-4909-a4a8-8d0f476490d0-kube-api-access-zsl9s" pod "6d668935-89fc-4909-a4a8-8d0f476490d0" (UID: "6d668935-89fc-4909-a4a8-8d0f476490d0"). InnerVolumeSpecName "kube-api-access-zsl9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:22.964235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.964210 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-metrics-certs" pod "6d668935-89fc-4909-a4a8-8d0f476490d0" (UID: "6d668935-89fc-4909-a4a8-8d0f476490d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:36:22.964318 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.964274 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-maas-api-tls" pod "6d668935-89fc-4909-a4a8-8d0f476490d0" (UID: "6d668935-89fc-4909-a4a8-8d0f476490d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:36:22.985952 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:22.985927 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:23.063837 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:23.063743 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:36:23.063837 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:23.063788 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsl9s\" (UniqueName: \"kubernetes.io/projected/6d668935-89fc-4909-a4a8-8d0f476490d0-kube-api-access-zsl9s\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:36:23.063837 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:23.063804 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d668935-89fc-4909-a4a8-8d0f476490d0-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:36:23.594812 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:23.594777 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:23.729637 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:23.729606 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" Nov 02 08:36:23.729785 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:23.729641 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt" event={"ID":"6d668935-89fc-4909-a4a8-8d0f476490d0","Type":"ContainerDied","Data":"b4755c386e82075525efbe6bd50de4384abcf6b3b2904c6d6bf2fe223231a919"} Nov 02 08:36:23.729785 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:23.729698 5899 scope.go:122] "RemoveContainer" containerID="e0820b5150c23a7804a82f837f50806e8f9875010e01bbeeb88af3d8b648ebd9" Nov 02 08:36:23.784498 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:23.784462 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt"] Nov 02 08:36:23.796372 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:23.796340 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bb4907-5c8595465b-fnhtt"] Nov 02 08:36:23.819025 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:23.818995 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d668935-89fc-4909-a4a8-8d0f476490d0" path="/var/lib/kubelet/pods/6d668935-89fc-4909-a4a8-8d0f476490d0/volumes" Nov 02 08:36:25.305343 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:25.305308 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:27.048129 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:27.048088 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:27.910078 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:27.910041 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:28.678294 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:28.678263 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:28.808332 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:28.808295 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:29.348232 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:29.348185 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:29.714660 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:29.714570 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:30.515789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:30.515753 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:30.648807 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:30.648762 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" podUID="b660cf56-4367-4815-ae30-acbca8e0ce0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 08:36:30.694942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:30.694910 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:31.178260 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:31.178228 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:31.318179 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:31.318126 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:31.978848 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:31.978801 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:32.405237 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:32.405099 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:32.804482 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:32.804452 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:33.213565 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:33.213472 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:33.805412 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:33.805378 5899 generic.go:356] "Generic (PLEG): container finished" podID="b660cf56-4367-4815-ae30-acbca8e0ce0b" containerID="3c127c305c2e7345b79252321ace8bc40d53b808a5767738717041dbcba28951" exitCode=0 Nov 02 08:36:33.805794 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:33.805435 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" event={"ID":"b660cf56-4367-4815-ae30-acbca8e0ce0b","Type":"ContainerDied","Data":"3c127c305c2e7345b79252321ace8bc40d53b808a5767738717041dbcba28951"} Nov 02 08:36:33.841479 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:33.841406 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" Nov 02 08:36:33.895765 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:33.895724 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b660cf56-4367-4815-ae30-acbca8e0ce0b-kube-api-access-w8xh6\" (UniqueName: \"kubernetes.io/projected/b660cf56-4367-4815-ae30-acbca8e0ce0b-kube-api-access-w8xh6\") pod \"b660cf56-4367-4815-ae30-acbca8e0ce0b\" (UID: \"b660cf56-4367-4815-ae30-acbca8e0ce0b\") " Nov 02 08:36:33.895958 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:33.895896 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b660cf56-4367-4815-ae30-acbca8e0ce0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b660cf56-4367-4815-ae30-acbca8e0ce0b-plugins-config-volume\") pod \"b660cf56-4367-4815-ae30-acbca8e0ce0b\" (UID: \"b660cf56-4367-4815-ae30-acbca8e0ce0b\") " Nov 02 08:36:33.896405 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:33.896374 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b660cf56-4367-4815-ae30-acbca8e0ce0b-plugins-config-volume" pod "b660cf56-4367-4815-ae30-acbca8e0ce0b" (UID: "b660cf56-4367-4815-ae30-acbca8e0ce0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:36:33.899392 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:33.899366 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b660cf56-4367-4815-ae30-acbca8e0ce0b-kube-api-access-w8xh6" pod "b660cf56-4367-4815-ae30-acbca8e0ce0b" (UID: "b660cf56-4367-4815-ae30-acbca8e0ce0b"). InnerVolumeSpecName "kube-api-access-w8xh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:33.997579 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:33.997537 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8xh6\" (UniqueName: \"kubernetes.io/projected/b660cf56-4367-4815-ae30-acbca8e0ce0b-kube-api-access-w8xh6\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:36:33.997579 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:33.997570 5899 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b660cf56-4367-4815-ae30-acbca8e0ce0b-plugins-config-volume\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:36:34.611649 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:34.611610 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:34.812177 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:34.812138 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" Nov 02 08:36:34.812690 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:34.812188 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz" event={"ID":"b660cf56-4367-4815-ae30-acbca8e0ce0b","Type":"ContainerDied","Data":"e92307d1971b872dc87dbc512b2f8a8a8080c2e3313377cc67f9cce16a329aea"} Nov 02 08:36:34.812690 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:34.812238 5899 scope.go:122] "RemoveContainer" containerID="3c127c305c2e7345b79252321ace8bc40d53b808a5767738717041dbcba28951" Nov 02 08:36:34.865891 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:34.865808 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz"] Nov 02 08:36:34.878963 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:34.878931 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bb4907-6c88cfdd4c-vjrcz"] Nov 02 08:36:35.818639 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:35.818606 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b660cf56-4367-4815-ae30-acbca8e0ce0b" path="/var/lib/kubelet/pods/b660cf56-4367-4815-ae30-acbca8e0ce0b/volumes" Nov 02 08:36:36.536000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:36.535966 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:36.678318 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:36.678284 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:37.635822 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:37.635742 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:38.277064 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:38.277025 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:38.463099 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:38.463058 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:38.729587 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:38.729556 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:40.196292 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:40.196256 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:40.659484 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:40.659450 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:40.799550 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:40.799511 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:41.996984 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:41.996936 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:43.141983 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:36:43.141938 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:36:43.601361 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:43.601317 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:43.755205 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:43.755151 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:43.889433 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:43.889349 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:45.154488 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.154446 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm"] Nov 02 08:36:45.155026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.154761 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" podUID="f3a71828-ac01-499e-90d3-6bc754414014" containerName="maas-api" containerID="cri-o://aff3e890ec3ccd14129533981db0d030cbf3825864ab19720ed0c00386e4250f" gracePeriod=30 Nov 02 08:36:45.238880 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.238835 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:45.381007 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.380968 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:45.631593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.631565 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:36:45.715445 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.715365 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:45.731529 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.731498 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-maas-api-tls\") pod \"f3a71828-ac01-499e-90d3-6bc754414014\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " Nov 02 08:36:45.731717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.731554 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3a71828-ac01-499e-90d3-6bc754414014-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a71828-ac01-499e-90d3-6bc754414014-openshift-service-ca\") pod \"f3a71828-ac01-499e-90d3-6bc754414014\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " Nov 02 08:36:45.731717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.731672 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-metrics-certs\") pod \"f3a71828-ac01-499e-90d3-6bc754414014\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " Nov 02 08:36:45.731840 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.731720 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3a71828-ac01-499e-90d3-6bc754414014-kube-api-access-zbn2k\" (UniqueName: \"kubernetes.io/projected/f3a71828-ac01-499e-90d3-6bc754414014-kube-api-access-zbn2k\") pod \"f3a71828-ac01-499e-90d3-6bc754414014\" (UID: \"f3a71828-ac01-499e-90d3-6bc754414014\") " Nov 02 08:36:45.731984 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.731953 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a71828-ac01-499e-90d3-6bc754414014-openshift-service-ca" pod "f3a71828-ac01-499e-90d3-6bc754414014" (UID: "f3a71828-ac01-499e-90d3-6bc754414014"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:36:45.732329 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.732304 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a71828-ac01-499e-90d3-6bc754414014-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:36:45.733859 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.733818 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-metrics-certs" pod "f3a71828-ac01-499e-90d3-6bc754414014" (UID: "f3a71828-ac01-499e-90d3-6bc754414014"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:36:45.733991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.733888 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a71828-ac01-499e-90d3-6bc754414014-kube-api-access-zbn2k" pod "f3a71828-ac01-499e-90d3-6bc754414014" (UID: "f3a71828-ac01-499e-90d3-6bc754414014"). InnerVolumeSpecName "kube-api-access-zbn2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:36:45.733991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.733967 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-maas-api-tls" pod "f3a71828-ac01-499e-90d3-6bc754414014" (UID: "f3a71828-ac01-499e-90d3-6bc754414014"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:36:45.832934 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.832902 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:36:45.832934 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.832934 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a71828-ac01-499e-90d3-6bc754414014-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:36:45.833111 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.832948 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbn2k\" (UniqueName: \"kubernetes.io/projected/f3a71828-ac01-499e-90d3-6bc754414014-kube-api-access-zbn2k\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:36:45.879625 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.879589 5899 generic.go:356] "Generic (PLEG): container finished" podID="f3a71828-ac01-499e-90d3-6bc754414014" containerID="aff3e890ec3ccd14129533981db0d030cbf3825864ab19720ed0c00386e4250f" exitCode=0 Nov 02 08:36:45.879800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.879639 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" Nov 02 08:36:45.879800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.879671 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" event={"ID":"f3a71828-ac01-499e-90d3-6bc754414014","Type":"ContainerDied","Data":"aff3e890ec3ccd14129533981db0d030cbf3825864ab19720ed0c00386e4250f"} Nov 02 08:36:45.879800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.879727 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm" event={"ID":"f3a71828-ac01-499e-90d3-6bc754414014","Type":"ContainerDied","Data":"90593831659d35c2d96886a41ccd0d41125b393b063fe3980989903038a49ef8"} Nov 02 08:36:45.879800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.879749 5899 scope.go:122] "RemoveContainer" containerID="aff3e890ec3ccd14129533981db0d030cbf3825864ab19720ed0c00386e4250f" Nov 02 08:36:45.889494 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.889470 5899 scope.go:122] "RemoveContainer" containerID="aff3e890ec3ccd14129533981db0d030cbf3825864ab19720ed0c00386e4250f" Nov 02 08:36:45.889815 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:36:45.889795 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff3e890ec3ccd14129533981db0d030cbf3825864ab19720ed0c00386e4250f\": container with ID starting with aff3e890ec3ccd14129533981db0d030cbf3825864ab19720ed0c00386e4250f not found: ID does not exist" containerID="aff3e890ec3ccd14129533981db0d030cbf3825864ab19720ed0c00386e4250f" Nov 02 08:36:45.889883 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.889829 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff3e890ec3ccd14129533981db0d030cbf3825864ab19720ed0c00386e4250f"} err="failed to get container status \"aff3e890ec3ccd14129533981db0d030cbf3825864ab19720ed0c00386e4250f\": rpc error: code = NotFound desc = could not find container \"aff3e890ec3ccd14129533981db0d030cbf3825864ab19720ed0c00386e4250f\": container with ID starting with aff3e890ec3ccd14129533981db0d030cbf3825864ab19720ed0c00386e4250f not found: ID does not exist" Nov 02 08:36:45.934480 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.934441 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm"] Nov 02 08:36:45.936308 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.936285 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3d8293-5fb9fd7ffb-nv6nm"] Nov 02 08:36:45.974790 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:45.974700 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:46.602001 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:46.601965 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:47.808062 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:47.808024 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:47.818991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:47.818960 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3a71828-ac01-499e-90d3-6bc754414014" path="/var/lib/kubelet/pods/f3a71828-ac01-499e-90d3-6bc754414014/volumes" Nov 02 08:36:49.303485 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:49.303449 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:50.012644 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:50.012600 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:50.246428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:50.246393 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:50.382544 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:50.382449 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:51.081123 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:51.081089 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:51.472542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:51.472461 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:51.933807 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:51.933769 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:52.067248 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:52.067204 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:52.274613 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:52.274578 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:52.418826 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:52.418789 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:52.964051 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:52.964001 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:53.325229 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:53.325196 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:53.577419 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:53.577336 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:53.814857 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:53.814822 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:54.066941 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:54.066907 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:55.603303 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:55.603267 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:56.983687 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:56.983651 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx"] Nov 02 08:36:57.127996 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:57.127957 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:57.263555 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:57.263465 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:57.875476 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:57.875436 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:58.041244 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:58.041204 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:58.752096 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:58.752063 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:58.894153 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:58.894110 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:36:59.715371 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:36:59.715330 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:00.150927 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:00.150886 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:00.282952 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:00.282920 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:01.231102 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:01.231063 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:01.592112 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:01.592073 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:03.071389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:03.071356 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:03.208588 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:03.208556 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:03.359371 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:03.359292 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:03.503992 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:03.503947 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:03.781478 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:03.781441 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:04.186944 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:04.186856 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:04.957020 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:04.956988 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:05.002888 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.002852 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb"] Nov 02 08:37:05.003099 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.003053 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" podUID="9e264650-6da9-478e-b892-45afa95918e9" containerName="maas-api" containerID="cri-o://c93e4dd27efe3d09e0e0aaeac3db5d8717c34f528b0322f6aaacdf2fe01c6906" gracePeriod=30 Nov 02 08:37:05.101553 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.101522 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:05.234341 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.234308 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:05.500255 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.500187 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:37:05.573376 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.573334 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e264650-6da9-478e-b892-45afa95918e9-kube-api-access-mbpqp\" (UniqueName: \"kubernetes.io/projected/9e264650-6da9-478e-b892-45afa95918e9-kube-api-access-mbpqp\") pod \"9e264650-6da9-478e-b892-45afa95918e9\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " Nov 02 08:37:05.573555 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.573393 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e264650-6da9-478e-b892-45afa95918e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e264650-6da9-478e-b892-45afa95918e9-openshift-service-ca\") pod \"9e264650-6da9-478e-b892-45afa95918e9\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " Nov 02 08:37:05.573555 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.573519 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-metrics-certs\") pod \"9e264650-6da9-478e-b892-45afa95918e9\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " Nov 02 08:37:05.573671 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.573654 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-maas-api-tls\") pod \"9e264650-6da9-478e-b892-45afa95918e9\" (UID: \"9e264650-6da9-478e-b892-45afa95918e9\") " Nov 02 08:37:05.573816 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.573782 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e264650-6da9-478e-b892-45afa95918e9-openshift-service-ca" pod "9e264650-6da9-478e-b892-45afa95918e9" (UID: "9e264650-6da9-478e-b892-45afa95918e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:05.574124 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.574101 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e264650-6da9-478e-b892-45afa95918e9-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:05.576423 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.576394 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e264650-6da9-478e-b892-45afa95918e9-kube-api-access-mbpqp" pod "9e264650-6da9-478e-b892-45afa95918e9" (UID: "9e264650-6da9-478e-b892-45afa95918e9"). InnerVolumeSpecName "kube-api-access-mbpqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:05.576561 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.576432 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-metrics-certs" pod "9e264650-6da9-478e-b892-45afa95918e9" (UID: "9e264650-6da9-478e-b892-45afa95918e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:05.576561 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.576532 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-maas-api-tls" pod "9e264650-6da9-478e-b892-45afa95918e9" (UID: "9e264650-6da9-478e-b892-45afa95918e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:05.675604 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.675562 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbpqp\" (UniqueName: \"kubernetes.io/projected/9e264650-6da9-478e-b892-45afa95918e9-kube-api-access-mbpqp\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:05.675604 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.675600 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:05.675604 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.675610 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e264650-6da9-478e-b892-45afa95918e9-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:05.816383 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.816303 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:05.995689 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.995654 5899 generic.go:356] "Generic (PLEG): container finished" podID="9e264650-6da9-478e-b892-45afa95918e9" containerID="c93e4dd27efe3d09e0e0aaeac3db5d8717c34f528b0322f6aaacdf2fe01c6906" exitCode=0 Nov 02 08:37:05.995878 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.995708 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" Nov 02 08:37:05.995878 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.995739 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" event={"ID":"9e264650-6da9-478e-b892-45afa95918e9","Type":"ContainerDied","Data":"c93e4dd27efe3d09e0e0aaeac3db5d8717c34f528b0322f6aaacdf2fe01c6906"} Nov 02 08:37:05.995878 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.995793 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb" event={"ID":"9e264650-6da9-478e-b892-45afa95918e9","Type":"ContainerDied","Data":"28867b7668d25a7bd7b43e8ecd0334b75fb95cf02b61dcf3c6c41bdea64a7fb4"} Nov 02 08:37:05.995878 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:05.995814 5899 scope.go:122] "RemoveContainer" containerID="c93e4dd27efe3d09e0e0aaeac3db5d8717c34f528b0322f6aaacdf2fe01c6906" Nov 02 08:37:06.005588 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:06.005444 5899 scope.go:122] "RemoveContainer" containerID="c93e4dd27efe3d09e0e0aaeac3db5d8717c34f528b0322f6aaacdf2fe01c6906" Nov 02 08:37:06.005773 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:37:06.005751 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93e4dd27efe3d09e0e0aaeac3db5d8717c34f528b0322f6aaacdf2fe01c6906\": container with ID starting with c93e4dd27efe3d09e0e0aaeac3db5d8717c34f528b0322f6aaacdf2fe01c6906 not found: ID does not exist" containerID="c93e4dd27efe3d09e0e0aaeac3db5d8717c34f528b0322f6aaacdf2fe01c6906" Nov 02 08:37:06.005838 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:06.005788 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93e4dd27efe3d09e0e0aaeac3db5d8717c34f528b0322f6aaacdf2fe01c6906"} err="failed to get container status \"c93e4dd27efe3d09e0e0aaeac3db5d8717c34f528b0322f6aaacdf2fe01c6906\": rpc error: code = NotFound desc = could not find container \"c93e4dd27efe3d09e0e0aaeac3db5d8717c34f528b0322f6aaacdf2fe01c6906\": container with ID starting with c93e4dd27efe3d09e0e0aaeac3db5d8717c34f528b0322f6aaacdf2fe01c6906 not found: ID does not exist" Nov 02 08:37:06.050265 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:06.050225 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb"] Nov 02 08:37:06.056527 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:06.056495 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a44352-b6854564c-sjpqb"] Nov 02 08:37:07.391018 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:07.390984 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:07.531294 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:07.531259 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:07.686718 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:07.686626 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:07.819545 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:07.819507 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e264650-6da9-478e-b892-45afa95918e9" path="/var/lib/kubelet/pods/9e264650-6da9-478e-b892-45afa95918e9/volumes" Nov 02 08:37:08.167987 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:08.167954 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:08.575889 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:08.575846 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:09.285140 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:09.285108 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:09.621123 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:09.621040 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:10.183341 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:10.183308 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:10.381211 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:10.381175 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:11.748614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:11.748521 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:12.005012 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.004922 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" containerID="cri-o://9ebe781a811a12d313a17b15991df29062078ba33f53f4fbe48ed41e7c11ffa6" gracePeriod=45 Nov 02 08:37:12.165224 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.165153 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused" Nov 02 08:37:12.267822 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.267756 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 08:37:12.346780 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.346733 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/908ba798-56ae-457f-a348-88ae00e00ed9-tls-certs\" (UniqueName: \"kubernetes.io/secret/908ba798-56ae-457f-a348-88ae00e00ed9-tls-certs\") pod \"908ba798-56ae-457f-a348-88ae00e00ed9\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " Nov 02 08:37:12.346975 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.346837 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-model-cache\") pod \"908ba798-56ae-457f-a348-88ae00e00ed9\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " Nov 02 08:37:12.346975 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.346914 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-tmp-dir\") pod \"908ba798-56ae-457f-a348-88ae00e00ed9\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " Nov 02 08:37:12.347116 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.346999 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-dshm\") pod \"908ba798-56ae-457f-a348-88ae00e00ed9\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " Nov 02 08:37:12.347198 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.347145 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908ba798-56ae-457f-a348-88ae00e00ed9-kube-api-access-swxks\" (UniqueName: \"kubernetes.io/projected/908ba798-56ae-457f-a348-88ae00e00ed9-kube-api-access-swxks\") pod \"908ba798-56ae-457f-a348-88ae00e00ed9\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " Nov 02 08:37:12.347269 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.347193 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-home\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-home\") pod \"908ba798-56ae-457f-a348-88ae00e00ed9\" (UID: \"908ba798-56ae-457f-a348-88ae00e00ed9\") " Nov 02 08:37:12.347417 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.347389 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-model-cache" pod "908ba798-56ae-457f-a348-88ae00e00ed9" (UID: "908ba798-56ae-457f-a348-88ae00e00ed9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:12.347675 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.347641 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-home" pod "908ba798-56ae-457f-a348-88ae00e00ed9" (UID: "908ba798-56ae-457f-a348-88ae00e00ed9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:12.347675 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.347659 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:12.348782 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.348744 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-tmp-dir" pod "908ba798-56ae-457f-a348-88ae00e00ed9" (UID: "908ba798-56ae-457f-a348-88ae00e00ed9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:12.350568 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.350544 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-dshm" pod "908ba798-56ae-457f-a348-88ae00e00ed9" (UID: "908ba798-56ae-457f-a348-88ae00e00ed9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:12.350568 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.350555 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908ba798-56ae-457f-a348-88ae00e00ed9-tls-certs" pod "908ba798-56ae-457f-a348-88ae00e00ed9" (UID: "908ba798-56ae-457f-a348-88ae00e00ed9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:12.350706 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.350683 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908ba798-56ae-457f-a348-88ae00e00ed9-kube-api-access-swxks" pod "908ba798-56ae-457f-a348-88ae00e00ed9" (UID: "908ba798-56ae-457f-a348-88ae00e00ed9"). InnerVolumeSpecName "kube-api-access-swxks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:12.448506 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.448468 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:12.448506 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.448500 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:12.448506 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.448510 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swxks\" (UniqueName: \"kubernetes.io/projected/908ba798-56ae-457f-a348-88ae00e00ed9-kube-api-access-swxks\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:12.448838 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.448519 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908ba798-56ae-457f-a348-88ae00e00ed9-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:12.448838 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:12.448529 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908ba798-56ae-457f-a348-88ae00e00ed9-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:13.035186 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:13.035128 5899 generic.go:356] "Generic (PLEG): container finished" podID="908ba798-56ae-457f-a348-88ae00e00ed9" containerID="9ebe781a811a12d313a17b15991df29062078ba33f53f4fbe48ed41e7c11ffa6" exitCode=0 Nov 02 08:37:13.035186 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:13.035188 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" Nov 02 08:37:13.035673 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:13.035213 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" event={"ID":"908ba798-56ae-457f-a348-88ae00e00ed9","Type":"ContainerDied","Data":"9ebe781a811a12d313a17b15991df29062078ba33f53f4fbe48ed41e7c11ffa6"} Nov 02 08:37:13.035673 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:13.035274 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx" event={"ID":"908ba798-56ae-457f-a348-88ae00e00ed9","Type":"ContainerDied","Data":"baa84e83a28b1a81c22ba2ae57cb301982543be174f51dacfa94bcadd535719b"} Nov 02 08:37:13.035673 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:13.035294 5899 scope.go:122] "RemoveContainer" containerID="9ebe781a811a12d313a17b15991df29062078ba33f53f4fbe48ed41e7c11ffa6" Nov 02 08:37:13.054888 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:13.054866 5899 scope.go:122] "RemoveContainer" containerID="9ebe781a811a12d313a17b15991df29062078ba33f53f4fbe48ed41e7c11ffa6" Nov 02 08:37:13.055272 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:37:13.055244 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ebe781a811a12d313a17b15991df29062078ba33f53f4fbe48ed41e7c11ffa6\": container with ID starting with 9ebe781a811a12d313a17b15991df29062078ba33f53f4fbe48ed41e7c11ffa6 not found: ID does not exist" containerID="9ebe781a811a12d313a17b15991df29062078ba33f53f4fbe48ed41e7c11ffa6" Nov 02 08:37:13.055374 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:13.055279 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ebe781a811a12d313a17b15991df29062078ba33f53f4fbe48ed41e7c11ffa6"} err="failed to get container status \"9ebe781a811a12d313a17b15991df29062078ba33f53f4fbe48ed41e7c11ffa6\": rpc error: code = NotFound desc = could not find container \"9ebe781a811a12d313a17b15991df29062078ba33f53f4fbe48ed41e7c11ffa6\": container with ID starting with 9ebe781a811a12d313a17b15991df29062078ba33f53f4fbe48ed41e7c11ffa6 not found: ID does not exist" Nov 02 08:37:13.090203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:13.090146 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx"] Nov 02 08:37:13.102377 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:13.102337 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-354660/distinct-w3-354660-kserve-547796d646-hkctx"] Nov 02 08:37:13.141378 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:37:13.141342 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:37:13.387505 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:13.387429 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:13.534743 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:13.534696 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:13.819856 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:13.819826 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" path="/var/lib/kubelet/pods/908ba798-56ae-457f-a348-88ae00e00ed9/volumes" Nov 02 08:37:14.098603 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:14.098520 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:14.302137 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:14.302105 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:14.724756 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:14.724726 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:14.865822 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:14.865789 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:15.135049 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:15.135010 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:15.812513 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:15.812476 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:15.951706 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:15.951645 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:17.150837 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.150800 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4"] Nov 02 08:37:17.151403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.151112 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" podUID="b0284bc4-0507-4252-8aee-2ef6448316e8" containerName="maas-api" containerID="cri-o://9d5a46a8014c0e9f7ed63ad28ce1a9d571c6795634cb787771280b01736337c1" gracePeriod=30 Nov 02 08:37:17.243994 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.243961 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:17.443208 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.443179 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 08:37:17.508839 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.508799 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0284bc4-0507-4252-8aee-2ef6448316e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0284bc4-0507-4252-8aee-2ef6448316e8-openshift-service-ca\") pod \"b0284bc4-0507-4252-8aee-2ef6448316e8\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " Nov 02 08:37:17.508839 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.508841 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-metrics-certs\") pod \"b0284bc4-0507-4252-8aee-2ef6448316e8\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " Nov 02 08:37:17.509103 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.508875 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0284bc4-0507-4252-8aee-2ef6448316e8-kube-api-access-45s4q\" (UniqueName: \"kubernetes.io/projected/b0284bc4-0507-4252-8aee-2ef6448316e8-kube-api-access-45s4q\") pod \"b0284bc4-0507-4252-8aee-2ef6448316e8\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " Nov 02 08:37:17.509103 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.509016 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-maas-api-tls\") pod \"b0284bc4-0507-4252-8aee-2ef6448316e8\" (UID: \"b0284bc4-0507-4252-8aee-2ef6448316e8\") " Nov 02 08:37:17.509317 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.509292 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0284bc4-0507-4252-8aee-2ef6448316e8-openshift-service-ca" pod "b0284bc4-0507-4252-8aee-2ef6448316e8" (UID: "b0284bc4-0507-4252-8aee-2ef6448316e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:17.509673 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.509651 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0284bc4-0507-4252-8aee-2ef6448316e8-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:17.511123 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.511097 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-metrics-certs" pod "b0284bc4-0507-4252-8aee-2ef6448316e8" (UID: "b0284bc4-0507-4252-8aee-2ef6448316e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:17.511610 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.511583 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-maas-api-tls" pod "b0284bc4-0507-4252-8aee-2ef6448316e8" (UID: "b0284bc4-0507-4252-8aee-2ef6448316e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:17.511610 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.511601 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0284bc4-0507-4252-8aee-2ef6448316e8-kube-api-access-45s4q" pod "b0284bc4-0507-4252-8aee-2ef6448316e8" (UID: "b0284bc4-0507-4252-8aee-2ef6448316e8"). InnerVolumeSpecName "kube-api-access-45s4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:17.532943 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.532671 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq"] Nov 02 08:37:17.532943 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.532881 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" podUID="31d07d4a-0d71-4586-95c2-4b67389ec4bd" containerName="maas-api" containerID="cri-o://83c72e61ee62f6bf096b66c3a202f5e6c27c94741934cdff64c800fe2bfe9461" gracePeriod=30 Nov 02 08:37:17.611512 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.611469 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:17.611512 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.611510 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45s4q\" (UniqueName: \"kubernetes.io/projected/b0284bc4-0507-4252-8aee-2ef6448316e8-kube-api-access-45s4q\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:17.611779 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.611524 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0284bc4-0507-4252-8aee-2ef6448316e8-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:17.615893 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.615855 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:17.797247 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.795795 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:37:17.915221 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.915156 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d07d4a-0d71-4586-95c2-4b67389ec4bd-kube-api-access-8jrbk\" (UniqueName: \"kubernetes.io/projected/31d07d4a-0d71-4586-95c2-4b67389ec4bd-kube-api-access-8jrbk\") pod \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " Nov 02 08:37:17.915427 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.915239 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-maas-api-tls\") pod \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " Nov 02 08:37:17.915427 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.915273 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-metrics-certs\") pod \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " Nov 02 08:37:17.915427 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.915300 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d07d4a-0d71-4586-95c2-4b67389ec4bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d07d4a-0d71-4586-95c2-4b67389ec4bd-openshift-service-ca\") pod \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\" (UID: \"31d07d4a-0d71-4586-95c2-4b67389ec4bd\") " Nov 02 08:37:17.915730 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.915694 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d07d4a-0d71-4586-95c2-4b67389ec4bd-openshift-service-ca" pod "31d07d4a-0d71-4586-95c2-4b67389ec4bd" (UID: "31d07d4a-0d71-4586-95c2-4b67389ec4bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:17.917429 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.917400 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d07d4a-0d71-4586-95c2-4b67389ec4bd-kube-api-access-8jrbk" pod "31d07d4a-0d71-4586-95c2-4b67389ec4bd" (UID: "31d07d4a-0d71-4586-95c2-4b67389ec4bd"). InnerVolumeSpecName "kube-api-access-8jrbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:17.917429 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.917420 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-metrics-certs" pod "31d07d4a-0d71-4586-95c2-4b67389ec4bd" (UID: "31d07d4a-0d71-4586-95c2-4b67389ec4bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:17.917579 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:17.917433 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-maas-api-tls" pod "31d07d4a-0d71-4586-95c2-4b67389ec4bd" (UID: "31d07d4a-0d71-4586-95c2-4b67389ec4bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:18.016987 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.016948 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jrbk\" (UniqueName: \"kubernetes.io/projected/31d07d4a-0d71-4586-95c2-4b67389ec4bd-kube-api-access-8jrbk\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:18.016987 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.016982 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:18.016987 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.016993 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31d07d4a-0d71-4586-95c2-4b67389ec4bd-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:18.017276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.017002 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d07d4a-0d71-4586-95c2-4b67389ec4bd-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:18.067364 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.067325 5899 generic.go:356] "Generic (PLEG): container finished" podID="b0284bc4-0507-4252-8aee-2ef6448316e8" containerID="9d5a46a8014c0e9f7ed63ad28ce1a9d571c6795634cb787771280b01736337c1" exitCode=0 Nov 02 08:37:18.067520 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.067372 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" Nov 02 08:37:18.067520 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.067411 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" event={"ID":"b0284bc4-0507-4252-8aee-2ef6448316e8","Type":"ContainerDied","Data":"9d5a46a8014c0e9f7ed63ad28ce1a9d571c6795634cb787771280b01736337c1"} Nov 02 08:37:18.067520 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.067479 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4" event={"ID":"b0284bc4-0507-4252-8aee-2ef6448316e8","Type":"ContainerDied","Data":"a45732457bda392f288d47de2bb543691684d8956fd9cf593faa3b080efa29b0"} Nov 02 08:37:18.067520 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.067515 5899 scope.go:122] "RemoveContainer" containerID="9d5a46a8014c0e9f7ed63ad28ce1a9d571c6795634cb787771280b01736337c1" Nov 02 08:37:18.069058 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.069033 5899 generic.go:356] "Generic (PLEG): container finished" podID="31d07d4a-0d71-4586-95c2-4b67389ec4bd" containerID="83c72e61ee62f6bf096b66c3a202f5e6c27c94741934cdff64c800fe2bfe9461" exitCode=0 Nov 02 08:37:18.069198 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.069083 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" Nov 02 08:37:18.069298 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.069152 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" event={"ID":"31d07d4a-0d71-4586-95c2-4b67389ec4bd","Type":"ContainerDied","Data":"83c72e61ee62f6bf096b66c3a202f5e6c27c94741934cdff64c800fe2bfe9461"} Nov 02 08:37:18.069527 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.069457 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq" event={"ID":"31d07d4a-0d71-4586-95c2-4b67389ec4bd","Type":"ContainerDied","Data":"31275362db5c85fc465e862a93ed7e44a90e17b00195d3a3148565102dd93c9b"} Nov 02 08:37:18.077899 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.077882 5899 scope.go:122] "RemoveContainer" containerID="9d5a46a8014c0e9f7ed63ad28ce1a9d571c6795634cb787771280b01736337c1" Nov 02 08:37:18.078170 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:37:18.078142 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5a46a8014c0e9f7ed63ad28ce1a9d571c6795634cb787771280b01736337c1\": container with ID starting with 9d5a46a8014c0e9f7ed63ad28ce1a9d571c6795634cb787771280b01736337c1 not found: ID does not exist" containerID="9d5a46a8014c0e9f7ed63ad28ce1a9d571c6795634cb787771280b01736337c1" Nov 02 08:37:18.078238 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.078193 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5a46a8014c0e9f7ed63ad28ce1a9d571c6795634cb787771280b01736337c1"} err="failed to get container status \"9d5a46a8014c0e9f7ed63ad28ce1a9d571c6795634cb787771280b01736337c1\": rpc error: code = NotFound desc = could not find container \"9d5a46a8014c0e9f7ed63ad28ce1a9d571c6795634cb787771280b01736337c1\": container with ID starting with 9d5a46a8014c0e9f7ed63ad28ce1a9d571c6795634cb787771280b01736337c1 not found: ID does not exist" Nov 02 08:37:18.078238 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.078209 5899 scope.go:122] "RemoveContainer" containerID="83c72e61ee62f6bf096b66c3a202f5e6c27c94741934cdff64c800fe2bfe9461" Nov 02 08:37:18.087030 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.087009 5899 scope.go:122] "RemoveContainer" containerID="83c72e61ee62f6bf096b66c3a202f5e6c27c94741934cdff64c800fe2bfe9461" Nov 02 08:37:18.087315 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:37:18.087296 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c72e61ee62f6bf096b66c3a202f5e6c27c94741934cdff64c800fe2bfe9461\": container with ID starting with 83c72e61ee62f6bf096b66c3a202f5e6c27c94741934cdff64c800fe2bfe9461 not found: ID does not exist" containerID="83c72e61ee62f6bf096b66c3a202f5e6c27c94741934cdff64c800fe2bfe9461" Nov 02 08:37:18.087372 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.087321 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c72e61ee62f6bf096b66c3a202f5e6c27c94741934cdff64c800fe2bfe9461"} err="failed to get container status \"83c72e61ee62f6bf096b66c3a202f5e6c27c94741934cdff64c800fe2bfe9461\": rpc error: code = NotFound desc = could not find container \"83c72e61ee62f6bf096b66c3a202f5e6c27c94741934cdff64c800fe2bfe9461\": container with ID starting with 83c72e61ee62f6bf096b66c3a202f5e6c27c94741934cdff64c800fe2bfe9461 not found: ID does not exist" Nov 02 08:37:18.121514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.121479 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4"] Nov 02 08:37:18.125231 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.125202 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-354660-5bb9cc844f-2jfk4"] Nov 02 08:37:18.160782 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.160748 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq"] Nov 02 08:37:18.164362 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.164335 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea9556-57d955d96b-vkxvq"] Nov 02 08:37:18.380477 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:18.380390 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:19.819807 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:19.819775 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d07d4a-0d71-4586-95c2-4b67389ec4bd" path="/var/lib/kubelet/pods/31d07d4a-0d71-4586-95c2-4b67389ec4bd/volumes" Nov 02 08:37:19.820228 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:19.820208 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0284bc4-0507-4252-8aee-2ef6448316e8" path="/var/lib/kubelet/pods/b0284bc4-0507-4252-8aee-2ef6448316e8/volumes" Nov 02 08:37:20.175763 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:20.175685 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:20.326350 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:20.326311 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:20.458852 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:20.458766 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:20.942244 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:20.942189 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:21.333083 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:21.333052 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:21.786233 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:21.786143 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:22.257576 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:22.257539 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:22.644002 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:22.643895 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:22.809059 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:22.809024 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:23.438456 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:23.438420 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:23.833787 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:23.833751 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:24.300611 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:24.300578 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:24.434985 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:24.434927 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:24.567703 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:24.567611 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:26.301889 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:26.301853 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:26.712923 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:26.712838 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz"] Nov 02 08:37:26.725913 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:26.725880 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc"] Nov 02 08:37:26.737671 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:26.737645 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt"] Nov 02 08:37:26.883911 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:26.883854 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:27.816281 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:27.816246 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:28.098276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:28.098149 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:30.026436 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:30.026401 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:31.087354 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:31.087322 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:31.222141 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:31.222100 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:31.417971 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:31.417885 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:31.964232 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:31.964188 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:34.003479 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:34.003440 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:34.811275 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:34.811232 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:34.950662 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:34.950623 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:35.216109 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.215995 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:35.358634 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.358601 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:35.464037 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.463997 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt"] Nov 02 08:37:35.464236 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.464214 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" podUID="027801e5-b984-456c-a118-953eaf824988" containerName="maas-api" containerID="cri-o://6e0cbcfc819113643143bb74fb93a4daf6cf8ded472e6ad8dfe2297a9934bc67" gracePeriod=30 Nov 02 08:37:35.502137 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.502084 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:35.647823 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.645978 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:35.734584 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.734542 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:37:35.812742 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.812706 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-maas-api-tls\") pod \"027801e5-b984-456c-a118-953eaf824988\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " Nov 02 08:37:35.812919 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.812758 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027801e5-b984-456c-a118-953eaf824988-kube-api-access-qj6vg\" (UniqueName: \"kubernetes.io/projected/027801e5-b984-456c-a118-953eaf824988-kube-api-access-qj6vg\") pod \"027801e5-b984-456c-a118-953eaf824988\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " Nov 02 08:37:35.812919 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.812810 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-metrics-certs\" (UniqueName: \"kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-metrics-certs\") pod \"027801e5-b984-456c-a118-953eaf824988\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " Nov 02 08:37:35.813048 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.813008 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027801e5-b984-456c-a118-953eaf824988-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/027801e5-b984-456c-a118-953eaf824988-openshift-service-ca\") pod \"027801e5-b984-456c-a118-953eaf824988\" (UID: \"027801e5-b984-456c-a118-953eaf824988\") " Nov 02 08:37:35.813479 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.813437 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027801e5-b984-456c-a118-953eaf824988-openshift-service-ca" pod "027801e5-b984-456c-a118-953eaf824988" (UID: "027801e5-b984-456c-a118-953eaf824988"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:35.815201 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.815149 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-metrics-certs" pod "027801e5-b984-456c-a118-953eaf824988" (UID: "027801e5-b984-456c-a118-953eaf824988"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:35.815540 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.815497 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027801e5-b984-456c-a118-953eaf824988-kube-api-access-qj6vg" pod "027801e5-b984-456c-a118-953eaf824988" (UID: "027801e5-b984-456c-a118-953eaf824988"). InnerVolumeSpecName "kube-api-access-qj6vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:35.815645 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.815580 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-maas-api-tls" pod "027801e5-b984-456c-a118-953eaf824988" (UID: "027801e5-b984-456c-a118-953eaf824988"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:35.918815 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.918774 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/027801e5-b984-456c-a118-953eaf824988-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:35.918815 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.918816 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:35.919130 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.918831 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj6vg\" (UniqueName: \"kubernetes.io/projected/027801e5-b984-456c-a118-953eaf824988-kube-api-access-qj6vg\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:35.919130 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:35.918849 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/027801e5-b984-456c-a118-953eaf824988-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:36.166465 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:36.166380 5899 generic.go:356] "Generic (PLEG): container finished" podID="027801e5-b984-456c-a118-953eaf824988" containerID="6e0cbcfc819113643143bb74fb93a4daf6cf8ded472e6ad8dfe2297a9934bc67" exitCode=0 Nov 02 08:37:36.166465 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:36.166424 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" Nov 02 08:37:36.166688 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:36.166424 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" event={"ID":"027801e5-b984-456c-a118-953eaf824988","Type":"ContainerDied","Data":"6e0cbcfc819113643143bb74fb93a4daf6cf8ded472e6ad8dfe2297a9934bc67"} Nov 02 08:37:36.166688 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:36.166571 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt" event={"ID":"027801e5-b984-456c-a118-953eaf824988","Type":"ContainerDied","Data":"4d1657f232fee710c8f67977d21d9a7ce4999a7b6ce49aa7b90853d67c9e7aee"} Nov 02 08:37:36.166688 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:36.166609 5899 scope.go:122] "RemoveContainer" containerID="6e0cbcfc819113643143bb74fb93a4daf6cf8ded472e6ad8dfe2297a9934bc67" Nov 02 08:37:36.178043 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:36.178023 5899 scope.go:122] "RemoveContainer" containerID="6e0cbcfc819113643143bb74fb93a4daf6cf8ded472e6ad8dfe2297a9934bc67" Nov 02 08:37:36.178422 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:37:36.178398 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e0cbcfc819113643143bb74fb93a4daf6cf8ded472e6ad8dfe2297a9934bc67\": container with ID starting with 6e0cbcfc819113643143bb74fb93a4daf6cf8ded472e6ad8dfe2297a9934bc67 not found: ID does not exist" containerID="6e0cbcfc819113643143bb74fb93a4daf6cf8ded472e6ad8dfe2297a9934bc67" Nov 02 08:37:36.178495 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:36.178432 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e0cbcfc819113643143bb74fb93a4daf6cf8ded472e6ad8dfe2297a9934bc67"} err="failed to get container status \"6e0cbcfc819113643143bb74fb93a4daf6cf8ded472e6ad8dfe2297a9934bc67\": rpc error: code = NotFound desc = could not find container \"6e0cbcfc819113643143bb74fb93a4daf6cf8ded472e6ad8dfe2297a9934bc67\": container with ID starting with 6e0cbcfc819113643143bb74fb93a4daf6cf8ded472e6ad8dfe2297a9934bc67 not found: ID does not exist" Nov 02 08:37:36.220513 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:36.220474 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt"] Nov 02 08:37:36.233479 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:36.233447 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-71a41c-699d9ccdcf-88fbt"] Nov 02 08:37:36.242243 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:36.242205 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:36.453566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:36.453430 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:37.822345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:37.822308 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027801e5-b984-456c-a118-953eaf824988" path="/var/lib/kubelet/pods/027801e5-b984-456c-a118-953eaf824988/volumes" Nov 02 08:37:38.417457 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:38.417422 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:38.744321 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:38.744243 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:39.138826 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:39.138738 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:39.301862 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:39.301830 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:39.444512 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:39.444428 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:40.061225 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:40.061187 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:40.423079 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:40.422995 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:40.623939 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:40.623908 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:40.894839 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:40.894804 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:41.734593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:41.734558 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" podUID="6725cf86-9321-401c-85b8-fba788e1e5c5" containerName="main" containerID="cri-o://3044a3b26c1c078c0d608965f519f7161b0e2677f3c755fbc8389d9282c1536f" gracePeriod=45 Nov 02 08:37:41.745565 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:41.745531 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" podUID="6725cf86-9321-401c-85b8-fba788e1e5c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": dial tcp 10.134.0.66:8000: connect: connection refused" Nov 02 08:37:41.746677 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:41.746628 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" podUID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" containerName="main" containerID="cri-o://6eb612ad0bec54e0d4043861ff391bbcfe53fb4eca204c976894325dd6728e05" gracePeriod=45 Nov 02 08:37:41.758801 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:41.758771 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" containerID="cri-o://0e85ee29cc7daadec8137eac868fd20cd6371c848f53f4a8832c8e755541cc34" gracePeriod=45 Nov 02 08:37:42.050307 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.050273 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 08:37:42.055288 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.055023 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 08:37:42.059371 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.059344 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 08:37:42.191218 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191181 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be63900-74ab-48d6-aee0-6071d5d2a305-kube-api-access-ljlxp\" (UniqueName: \"kubernetes.io/projected/9be63900-74ab-48d6-aee0-6071d5d2a305-kube-api-access-ljlxp\") pod \"9be63900-74ab-48d6-aee0-6071d5d2a305\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " Nov 02 08:37:42.191426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191239 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-home\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-home\") pod \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " Nov 02 08:37:42.191426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191293 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-home\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-home\") pod \"9be63900-74ab-48d6-aee0-6071d5d2a305\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " Nov 02 08:37:42.191426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191369 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-model-cache\") pod \"6725cf86-9321-401c-85b8-fba788e1e5c5\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " Nov 02 08:37:42.191426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191400 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6725cf86-9321-401c-85b8-fba788e1e5c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/6725cf86-9321-401c-85b8-fba788e1e5c5-tls-certs\") pod \"6725cf86-9321-401c-85b8-fba788e1e5c5\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " Nov 02 08:37:42.191672 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191487 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tmp-dir\") pod \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " Nov 02 08:37:42.191672 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191513 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-home" pod "d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" (UID: "d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:42.191672 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191527 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-kube-api-access-skqsz\" (UniqueName: \"kubernetes.io/projected/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-kube-api-access-skqsz\") pod \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " Nov 02 08:37:42.191672 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191564 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-home\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-home\") pod \"6725cf86-9321-401c-85b8-fba788e1e5c5\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " Nov 02 08:37:42.191672 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191605 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9be63900-74ab-48d6-aee0-6071d5d2a305-tls-certs\" (UniqueName: \"kubernetes.io/secret/9be63900-74ab-48d6-aee0-6071d5d2a305-tls-certs\") pod \"9be63900-74ab-48d6-aee0-6071d5d2a305\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " Nov 02 08:37:42.191672 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191645 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-dshm\") pod \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191693 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-dshm\") pod \"9be63900-74ab-48d6-aee0-6071d5d2a305\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191719 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-dshm\") pod \"6725cf86-9321-401c-85b8-fba788e1e5c5\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191745 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-model-cache\") pod \"9be63900-74ab-48d6-aee0-6071d5d2a305\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191775 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-tmp-dir\") pod \"9be63900-74ab-48d6-aee0-6071d5d2a305\" (UID: \"9be63900-74ab-48d6-aee0-6071d5d2a305\") " Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191823 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6725cf86-9321-401c-85b8-fba788e1e5c5-kube-api-access-f4wc9\" (UniqueName: \"kubernetes.io/projected/6725cf86-9321-401c-85b8-fba788e1e5c5-kube-api-access-f4wc9\") pod \"6725cf86-9321-401c-85b8-fba788e1e5c5\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191882 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-model-cache\") pod \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191912 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tls-certs\") pod \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\" (UID: \"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418\") " Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191957 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-tmp-dir\") pod \"6725cf86-9321-401c-85b8-fba788e1e5c5\" (UID: \"6725cf86-9321-401c-85b8-fba788e1e5c5\") " Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.192447 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191644 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-model-cache" pod "6725cf86-9321-401c-85b8-fba788e1e5c5" (UID: "6725cf86-9321-401c-85b8-fba788e1e5c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191800 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-home" pod "9be63900-74ab-48d6-aee0-6071d5d2a305" (UID: "9be63900-74ab-48d6-aee0-6071d5d2a305"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.191942 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tmp-dir" pod "d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" (UID: "d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.192156 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-home" pod "6725cf86-9321-401c-85b8-fba788e1e5c5" (UID: "6725cf86-9321-401c-85b8-fba788e1e5c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:42.192842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.192238 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-model-cache" pod "9be63900-74ab-48d6-aee0-6071d5d2a305" (UID: "9be63900-74ab-48d6-aee0-6071d5d2a305"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:42.193685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.192927 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-model-cache" pod "d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" (UID: "d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:42.193685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.193118 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-tmp-dir" pod "6725cf86-9321-401c-85b8-fba788e1e5c5" (UID: "6725cf86-9321-401c-85b8-fba788e1e5c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:42.193685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.193130 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-tmp-dir" pod "9be63900-74ab-48d6-aee0-6071d5d2a305" (UID: "9be63900-74ab-48d6-aee0-6071d5d2a305"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:42.195641 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.195608 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-dshm" pod "d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" (UID: "d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:42.196282 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.196246 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6725cf86-9321-401c-85b8-fba788e1e5c5-kube-api-access-f4wc9" pod "6725cf86-9321-401c-85b8-fba788e1e5c5" (UID: "6725cf86-9321-401c-85b8-fba788e1e5c5"). InnerVolumeSpecName "kube-api-access-f4wc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:42.196530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.196508 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tls-certs" pod "d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" (UID: "d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:42.197739 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.197693 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6725cf86-9321-401c-85b8-fba788e1e5c5-tls-certs" pod "6725cf86-9321-401c-85b8-fba788e1e5c5" (UID: "6725cf86-9321-401c-85b8-fba788e1e5c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:42.197739 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.197699 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be63900-74ab-48d6-aee0-6071d5d2a305-tls-certs" pod "9be63900-74ab-48d6-aee0-6071d5d2a305" (UID: "9be63900-74ab-48d6-aee0-6071d5d2a305"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:42.197739 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.197729 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-kube-api-access-skqsz" pod "d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" (UID: "d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418"). InnerVolumeSpecName "kube-api-access-skqsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:42.197926 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.197848 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be63900-74ab-48d6-aee0-6071d5d2a305-kube-api-access-ljlxp" pod "9be63900-74ab-48d6-aee0-6071d5d2a305" (UID: "9be63900-74ab-48d6-aee0-6071d5d2a305"). InnerVolumeSpecName "kube-api-access-ljlxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:42.199464 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.199442 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-dshm" pod "6725cf86-9321-401c-85b8-fba788e1e5c5" (UID: "6725cf86-9321-401c-85b8-fba788e1e5c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:42.199582 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.199523 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-dshm" pod "9be63900-74ab-48d6-aee0-6071d5d2a305" (UID: "9be63900-74ab-48d6-aee0-6071d5d2a305"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:37:42.221428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.221395 5899 generic.go:356] "Generic (PLEG): container finished" podID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerID="0e85ee29cc7daadec8137eac868fd20cd6371c848f53f4a8832c8e755541cc34" exitCode=0 Nov 02 08:37:42.221597 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.221454 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" Nov 02 08:37:42.221597 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.221487 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" event={"ID":"9be63900-74ab-48d6-aee0-6071d5d2a305","Type":"ContainerDied","Data":"0e85ee29cc7daadec8137eac868fd20cd6371c848f53f4a8832c8e755541cc34"} Nov 02 08:37:42.221597 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.221540 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt" event={"ID":"9be63900-74ab-48d6-aee0-6071d5d2a305","Type":"ContainerDied","Data":"0aff9e60bf8fbfb922762c94216e8c2061e4dd5b4baaf18fec8aa9112662723c"} Nov 02 08:37:42.221597 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.221570 5899 scope.go:122] "RemoveContainer" containerID="0e85ee29cc7daadec8137eac868fd20cd6371c848f53f4a8832c8e755541cc34" Nov 02 08:37:42.223197 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.223150 5899 generic.go:356] "Generic (PLEG): container finished" podID="6725cf86-9321-401c-85b8-fba788e1e5c5" containerID="3044a3b26c1c078c0d608965f519f7161b0e2677f3c755fbc8389d9282c1536f" exitCode=0 Nov 02 08:37:42.223347 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.223231 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" Nov 02 08:37:42.223347 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.223277 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" event={"ID":"6725cf86-9321-401c-85b8-fba788e1e5c5","Type":"ContainerDied","Data":"3044a3b26c1c078c0d608965f519f7161b0e2677f3c755fbc8389d9282c1536f"} Nov 02 08:37:42.223347 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.223307 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz" event={"ID":"6725cf86-9321-401c-85b8-fba788e1e5c5","Type":"ContainerDied","Data":"72cd0a71e456719d40c7d83c485cbbbdb43942920bcf219c60903a3b874b1fcd"} Nov 02 08:37:42.233991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.232451 5899 scope.go:122] "RemoveContainer" containerID="0e85ee29cc7daadec8137eac868fd20cd6371c848f53f4a8832c8e755541cc34" Nov 02 08:37:42.233991 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:37:42.233146 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e85ee29cc7daadec8137eac868fd20cd6371c848f53f4a8832c8e755541cc34\": container with ID starting with 0e85ee29cc7daadec8137eac868fd20cd6371c848f53f4a8832c8e755541cc34 not found: ID does not exist" containerID="0e85ee29cc7daadec8137eac868fd20cd6371c848f53f4a8832c8e755541cc34" Nov 02 08:37:42.233991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.233216 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e85ee29cc7daadec8137eac868fd20cd6371c848f53f4a8832c8e755541cc34"} err="failed to get container status \"0e85ee29cc7daadec8137eac868fd20cd6371c848f53f4a8832c8e755541cc34\": rpc error: code = NotFound desc = could not find container \"0e85ee29cc7daadec8137eac868fd20cd6371c848f53f4a8832c8e755541cc34\": container with ID starting with 0e85ee29cc7daadec8137eac868fd20cd6371c848f53f4a8832c8e755541cc34 not found: ID does not exist" Nov 02 08:37:42.233991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.233240 5899 scope.go:122] "RemoveContainer" containerID="3044a3b26c1c078c0d608965f519f7161b0e2677f3c755fbc8389d9282c1536f" Nov 02 08:37:42.234493 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.234473 5899 generic.go:356] "Generic (PLEG): container finished" podID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" containerID="6eb612ad0bec54e0d4043861ff391bbcfe53fb4eca204c976894325dd6728e05" exitCode=0 Nov 02 08:37:42.234575 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.234558 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" event={"ID":"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418","Type":"ContainerDied","Data":"6eb612ad0bec54e0d4043861ff391bbcfe53fb4eca204c976894325dd6728e05"} Nov 02 08:37:42.234643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.234574 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" Nov 02 08:37:42.234643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.234594 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc" event={"ID":"d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418","Type":"ContainerDied","Data":"81270840f7295d569f35be9ece694311bc11ab575c30d6d4f1af163ad6f24ed3"} Nov 02 08:37:42.244468 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.244182 5899 scope.go:122] "RemoveContainer" containerID="3044a3b26c1c078c0d608965f519f7161b0e2677f3c755fbc8389d9282c1536f" Nov 02 08:37:42.244552 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:37:42.244529 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3044a3b26c1c078c0d608965f519f7161b0e2677f3c755fbc8389d9282c1536f\": container with ID starting with 3044a3b26c1c078c0d608965f519f7161b0e2677f3c755fbc8389d9282c1536f not found: ID does not exist" containerID="3044a3b26c1c078c0d608965f519f7161b0e2677f3c755fbc8389d9282c1536f" Nov 02 08:37:42.244616 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.244563 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3044a3b26c1c078c0d608965f519f7161b0e2677f3c755fbc8389d9282c1536f"} err="failed to get container status \"3044a3b26c1c078c0d608965f519f7161b0e2677f3c755fbc8389d9282c1536f\": rpc error: code = NotFound desc = could not find container \"3044a3b26c1c078c0d608965f519f7161b0e2677f3c755fbc8389d9282c1536f\": container with ID starting with 3044a3b26c1c078c0d608965f519f7161b0e2677f3c755fbc8389d9282c1536f not found: ID does not exist" Nov 02 08:37:42.244616 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.244587 5899 scope.go:122] "RemoveContainer" containerID="6eb612ad0bec54e0d4043861ff391bbcfe53fb4eca204c976894325dd6728e05" Nov 02 08:37:42.258660 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.258635 5899 scope.go:122] "RemoveContainer" containerID="6eb612ad0bec54e0d4043861ff391bbcfe53fb4eca204c976894325dd6728e05" Nov 02 08:37:42.259026 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:37:42.259002 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb612ad0bec54e0d4043861ff391bbcfe53fb4eca204c976894325dd6728e05\": container with ID starting with 6eb612ad0bec54e0d4043861ff391bbcfe53fb4eca204c976894325dd6728e05 not found: ID does not exist" containerID="6eb612ad0bec54e0d4043861ff391bbcfe53fb4eca204c976894325dd6728e05" Nov 02 08:37:42.259132 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.259030 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb612ad0bec54e0d4043861ff391bbcfe53fb4eca204c976894325dd6728e05"} err="failed to get container status \"6eb612ad0bec54e0d4043861ff391bbcfe53fb4eca204c976894325dd6728e05\": rpc error: code = NotFound desc = could not find container \"6eb612ad0bec54e0d4043861ff391bbcfe53fb4eca204c976894325dd6728e05\": container with ID starting with 6eb612ad0bec54e0d4043861ff391bbcfe53fb4eca204c976894325dd6728e05 not found: ID does not exist" Nov 02 08:37:42.276194 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.276113 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz"] Nov 02 08:37:42.281072 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.281041 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/distinct-2-w4-9db802-kserve-7bdd7b699f-fg4fz"] Nov 02 08:37:42.293194 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293148 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293309 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293196 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skqsz\" (UniqueName: \"kubernetes.io/projected/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-kube-api-access-skqsz\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293309 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293213 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293309 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293228 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be63900-74ab-48d6-aee0-6071d5d2a305-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293309 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293242 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293309 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293253 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293309 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293264 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293309 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293277 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293309 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293291 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293309 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293304 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4wc9\" (UniqueName: \"kubernetes.io/projected/6725cf86-9321-401c-85b8-fba788e1e5c5-kube-api-access-f4wc9\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293316 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293330 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293342 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293354 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljlxp\" (UniqueName: \"kubernetes.io/projected/9be63900-74ab-48d6-aee0-6071d5d2a305-kube-api-access-ljlxp\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293366 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be63900-74ab-48d6-aee0-6071d5d2a305-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293379 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6725cf86-9321-401c-85b8-fba788e1e5c5-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.293614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.293391 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6725cf86-9321-401c-85b8-fba788e1e5c5-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:42.330711 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.330674 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt"] Nov 02 08:37:42.334775 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.334741 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/simulator-w4-9db802-kserve-65676b8468-vrmnt"] Nov 02 08:37:42.344561 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.344533 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:42.370868 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.370830 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc"] Nov 02 08:37:42.378194 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.378139 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/premium-w4-9db802-kserve-5496bf967c-zdmdc"] Nov 02 08:37:42.764426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:42.764390 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:43.140879 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:37:43.140797 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:37:43.291269 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:43.291234 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:43.454009 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:43.453930 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:43.818189 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:43.818123 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6725cf86-9321-401c-85b8-fba788e1e5c5" path="/var/lib/kubelet/pods/6725cf86-9321-401c-85b8-fba788e1e5c5/volumes" Nov 02 08:37:43.818800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:43.818777 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" path="/var/lib/kubelet/pods/9be63900-74ab-48d6-aee0-6071d5d2a305/volumes" Nov 02 08:37:43.819265 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:43.819244 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" path="/var/lib/kubelet/pods/d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418/volumes" Nov 02 08:37:44.124734 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:44.124647 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:45.925220 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:45.925181 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:45.926071 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:45.925756 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw"] Nov 02 08:37:45.937854 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:45.937815 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k"] Nov 02 08:37:46.086774 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:46.086732 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:46.620431 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:46.620387 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:47.707086 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:47.707048 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:48.897986 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:48.897947 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:49.408314 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:49.408277 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:51.199369 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:51.199316 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:52.561674 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:52.561637 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:52.657209 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:52.657169 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr"] Nov 02 08:37:52.657416 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:52.657393 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" podUID="c9c41bdc-daa2-4024-8042-cb6810f56988" containerName="maas-api" containerID="cri-o://5386296fc8130e1cd4d30b5d05778b991c34bca764429ee73e6196363fa1beb0" gracePeriod=30 Nov 02 08:37:52.716412 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:52.716370 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:52.867682 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:52.867644 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:52.906271 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:52.906244 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 08:37:53.013256 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.013224 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c41bdc-daa2-4024-8042-cb6810f56988-kube-api-access-bftss\" (UniqueName: \"kubernetes.io/projected/c9c41bdc-daa2-4024-8042-cb6810f56988-kube-api-access-bftss\") pod \"c9c41bdc-daa2-4024-8042-cb6810f56988\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " Nov 02 08:37:53.013448 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.013301 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-maas-api-tls\") pod \"c9c41bdc-daa2-4024-8042-cb6810f56988\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " Nov 02 08:37:53.013448 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.013331 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-metrics-certs\") pod \"c9c41bdc-daa2-4024-8042-cb6810f56988\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " Nov 02 08:37:53.013448 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.013361 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c41bdc-daa2-4024-8042-cb6810f56988-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c41bdc-daa2-4024-8042-cb6810f56988-openshift-service-ca\") pod \"c9c41bdc-daa2-4024-8042-cb6810f56988\" (UID: \"c9c41bdc-daa2-4024-8042-cb6810f56988\") " Nov 02 08:37:53.013791 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.013762 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c41bdc-daa2-4024-8042-cb6810f56988-openshift-service-ca" pod "c9c41bdc-daa2-4024-8042-cb6810f56988" (UID: "c9c41bdc-daa2-4024-8042-cb6810f56988"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:53.013931 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.013840 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c41bdc-daa2-4024-8042-cb6810f56988-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:53.015547 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.015524 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c41bdc-daa2-4024-8042-cb6810f56988-kube-api-access-bftss" pod "c9c41bdc-daa2-4024-8042-cb6810f56988" (UID: "c9c41bdc-daa2-4024-8042-cb6810f56988"). InnerVolumeSpecName "kube-api-access-bftss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:53.015547 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.015528 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-maas-api-tls" pod "c9c41bdc-daa2-4024-8042-cb6810f56988" (UID: "c9c41bdc-daa2-4024-8042-cb6810f56988"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:53.015718 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.015698 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-metrics-certs" pod "c9c41bdc-daa2-4024-8042-cb6810f56988" (UID: "c9c41bdc-daa2-4024-8042-cb6810f56988"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:53.114656 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.114569 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bftss\" (UniqueName: \"kubernetes.io/projected/c9c41bdc-daa2-4024-8042-cb6810f56988-kube-api-access-bftss\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:53.114656 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.114605 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:53.114656 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.114614 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c41bdc-daa2-4024-8042-cb6810f56988-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:53.133664 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.133630 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb"] Nov 02 08:37:53.134088 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134076 5899 cpu_manager.go:414] "containerMap: removing container" podUID="36f9a988-8ac6-4969-8e0e-f7a267376b20" containerName="maas-api" Nov 02 08:37:53.134138 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134089 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f9a988-8ac6-4969-8e0e-f7a267376b20" containerName="maas-api" Nov 02 08:37:53.134138 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134105 5899 cpu_manager.go:414] "containerMap: removing container" podUID="0915e267-30ab-478c-a749-0fb605399ee4" containerName="payload-processing" Nov 02 08:37:53.134138 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134111 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0915e267-30ab-478c-a749-0fb605399ee4" containerName="payload-processing" Nov 02 08:37:53.134138 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134123 5899 cpu_manager.go:414] "containerMap: removing container" podUID="053fdd58-57a4-42a5-9cd6-4891d05da186" containerName="main" Nov 02 08:37:53.134138 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134128 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="053fdd58-57a4-42a5-9cd6-4891d05da186" containerName="main" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134140 5899 cpu_manager.go:414] "containerMap: removing container" podUID="6d668935-89fc-4909-a4a8-8d0f476490d0" containerName="maas-api" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134145 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d668935-89fc-4909-a4a8-8d0f476490d0" containerName="maas-api" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134155 5899 cpu_manager.go:414] "containerMap: removing container" podUID="027801e5-b984-456c-a118-953eaf824988" containerName="maas-api" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134173 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027801e5-b984-456c-a118-953eaf824988" containerName="maas-api" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134195 5899 cpu_manager.go:414] "containerMap: removing container" podUID="c9c41bdc-daa2-4024-8042-cb6810f56988" containerName="maas-api" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134200 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c41bdc-daa2-4024-8042-cb6810f56988" containerName="maas-api" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134211 5899 cpu_manager.go:414] "containerMap: removing container" podUID="9e264650-6da9-478e-b892-45afa95918e9" containerName="maas-api" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134216 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e264650-6da9-478e-b892-45afa95918e9" containerName="maas-api" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134222 5899 cpu_manager.go:414] "containerMap: removing container" podUID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" containerName="main" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134227 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" containerName="main" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134238 5899 cpu_manager.go:414] "containerMap: removing container" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134242 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134251 5899 cpu_manager.go:414] "containerMap: removing container" podUID="b660cf56-4367-4815-ae30-acbca8e0ce0b" containerName="payload-pre-processing" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134256 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b660cf56-4367-4815-ae30-acbca8e0ce0b" containerName="payload-pre-processing" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134263 5899 cpu_manager.go:414] "containerMap: removing container" podUID="8f09bad9-8a3f-45ba-93c3-3bc09e153761" containerName="maas-api" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134267 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f09bad9-8a3f-45ba-93c3-3bc09e153761" containerName="maas-api" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134275 5899 cpu_manager.go:414] "containerMap: removing container" podUID="31d07d4a-0d71-4586-95c2-4b67389ec4bd" containerName="maas-api" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134280 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d07d4a-0d71-4586-95c2-4b67389ec4bd" containerName="maas-api" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134288 5899 cpu_manager.go:414] "containerMap: removing container" podUID="6725cf86-9321-401c-85b8-fba788e1e5c5" containerName="main" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134293 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6725cf86-9321-401c-85b8-fba788e1e5c5" containerName="main" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134302 5899 cpu_manager.go:414] "containerMap: removing container" podUID="73ee734d-db5b-4d7f-aff3-044bb4802ba1" containerName="maas-api" Nov 02 08:37:53.134301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134307 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ee734d-db5b-4d7f-aff3-044bb4802ba1" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134314 5899 cpu_manager.go:414] "containerMap: removing container" podUID="7bda83a4-b696-45c0-b3a2-9b03e75fe3d6" containerName="payload-pre-processing" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134319 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bda83a4-b696-45c0-b3a2-9b03e75fe3d6" containerName="payload-pre-processing" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134325 5899 cpu_manager.go:414] "containerMap: removing container" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134330 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134341 5899 cpu_manager.go:414] "containerMap: removing container" podUID="f3a71828-ac01-499e-90d3-6bc754414014" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134345 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3a71828-ac01-499e-90d3-6bc754414014" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134351 5899 cpu_manager.go:414] "containerMap: removing container" podUID="b0284bc4-0507-4252-8aee-2ef6448316e8" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134356 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0284bc4-0507-4252-8aee-2ef6448316e8" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134372 5899 cpu_manager.go:414] "containerMap: removing container" podUID="345e41d9-05b3-4795-a9fc-fce423798f9d" containerName="main" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134377 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345e41d9-05b3-4795-a9fc-fce423798f9d" containerName="main" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134438 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c41bdc-daa2-4024-8042-cb6810f56988" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134446 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e264650-6da9-478e-b892-45afa95918e9" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134453 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6bd9f32-cb7a-4a67-b7bb-cdf1d89ea418" containerName="main" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134459 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="908ba798-56ae-457f-a348-88ae00e00ed9" containerName="main" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134464 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="b660cf56-4367-4815-ae30-acbca8e0ce0b" containerName="payload-pre-processing" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134469 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f09bad9-8a3f-45ba-93c3-3bc09e153761" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134475 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d07d4a-0d71-4586-95c2-4b67389ec4bd" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134480 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="6725cf86-9321-401c-85b8-fba788e1e5c5" containerName="main" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134485 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ee734d-db5b-4d7f-aff3-044bb4802ba1" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134489 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bda83a4-b696-45c0-b3a2-9b03e75fe3d6" containerName="payload-pre-processing" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134495 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be63900-74ab-48d6-aee0-6071d5d2a305" containerName="main" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134502 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3a71828-ac01-499e-90d3-6bc754414014" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134507 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0284bc4-0507-4252-8aee-2ef6448316e8" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134514 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="345e41d9-05b3-4795-a9fc-fce423798f9d" containerName="main" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134520 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f9a988-8ac6-4969-8e0e-f7a267376b20" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134525 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="0915e267-30ab-478c-a749-0fb605399ee4" containerName="payload-processing" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134532 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="053fdd58-57a4-42a5-9cd6-4891d05da186" containerName="main" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134538 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d668935-89fc-4909-a4a8-8d0f476490d0" containerName="maas-api" Nov 02 08:37:53.134962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.134542 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="027801e5-b984-456c-a118-953eaf824988" containerName="maas-api" Nov 02 08:37:53.137783 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.137767 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" Nov 02 08:37:53.155051 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.155019 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:37:53.155248 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.155219 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\"" Nov 02 08:37:53.157388 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.157361 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb"] Nov 02 08:37:53.299619 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.299589 5899 generic.go:356] "Generic (PLEG): container finished" podID="c9c41bdc-daa2-4024-8042-cb6810f56988" containerID="5386296fc8130e1cd4d30b5d05778b991c34bca764429ee73e6196363fa1beb0" exitCode=0 Nov 02 08:37:53.299805 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.299636 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" Nov 02 08:37:53.299805 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.299670 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" event={"ID":"c9c41bdc-daa2-4024-8042-cb6810f56988","Type":"ContainerDied","Data":"5386296fc8130e1cd4d30b5d05778b991c34bca764429ee73e6196363fa1beb0"} Nov 02 08:37:53.299805 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.299724 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr" event={"ID":"c9c41bdc-daa2-4024-8042-cb6810f56988","Type":"ContainerDied","Data":"b1a810d22d9d046d962869ee32a17f03aebfc918271cbd3bac3de927aa90f765"} Nov 02 08:37:53.299805 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.299755 5899 scope.go:122] "RemoveContainer" containerID="5386296fc8130e1cd4d30b5d05778b991c34bca764429ee73e6196363fa1beb0" Nov 02 08:37:53.309342 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.309326 5899 scope.go:122] "RemoveContainer" containerID="5386296fc8130e1cd4d30b5d05778b991c34bca764429ee73e6196363fa1beb0" Nov 02 08:37:53.309614 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:37:53.309594 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5386296fc8130e1cd4d30b5d05778b991c34bca764429ee73e6196363fa1beb0\": container with ID starting with 5386296fc8130e1cd4d30b5d05778b991c34bca764429ee73e6196363fa1beb0 not found: ID does not exist" containerID="5386296fc8130e1cd4d30b5d05778b991c34bca764429ee73e6196363fa1beb0" Nov 02 08:37:53.309666 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.309622 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5386296fc8130e1cd4d30b5d05778b991c34bca764429ee73e6196363fa1beb0"} err="failed to get container status \"5386296fc8130e1cd4d30b5d05778b991c34bca764429ee73e6196363fa1beb0\": rpc error: code = NotFound desc = could not find container \"5386296fc8130e1cd4d30b5d05778b991c34bca764429ee73e6196363fa1beb0\": container with ID starting with 5386296fc8130e1cd4d30b5d05778b991c34bca764429ee73e6196363fa1beb0 not found: ID does not exist" Nov 02 08:37:53.316203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.316152 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df106b94-2rmvb\" (UID: \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" Nov 02 08:37:53.316310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.316220 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6bf9\" (UniqueName: \"kubernetes.io/projected/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-kube-api-access-p6bf9\") pod \"maas-api-revoke-sub-df106b94-2rmvb\" (UID: \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" Nov 02 08:37:53.316310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.316304 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df106b94-2rmvb\" (UID: \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" Nov 02 08:37:53.354344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.354318 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr"] Nov 02 08:37:53.356459 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.356437 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9db802-5bd6866847-b22rr"] Nov 02 08:37:53.417143 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.417068 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df106b94-2rmvb\" (UID: \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" Nov 02 08:37:53.417143 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.417104 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bf9\" (UniqueName: \"kubernetes.io/projected/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-kube-api-access-p6bf9\") pod \"maas-api-revoke-sub-df106b94-2rmvb\" (UID: \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" Nov 02 08:37:53.417370 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.417156 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df106b94-2rmvb\" (UID: \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" Nov 02 08:37:53.417910 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.417887 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df106b94-2rmvb\" (UID: \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" Nov 02 08:37:53.419477 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.419457 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df106b94-2rmvb\" (UID: \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" Nov 02 08:37:53.441257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.441237 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6bf9\" (UniqueName: \"kubernetes.io/projected/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-kube-api-access-p6bf9\") pod \"maas-api-revoke-sub-df106b94-2rmvb\" (UID: \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" Nov 02 08:37:53.452040 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.452019 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" Nov 02 08:37:53.755671 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.755646 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb"] Nov 02 08:37:53.757599 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:37:53.757572 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9420fdfa_e0b9_4a66_bbf6_1dbf986ad567.slice/crio-569d016ff428f7a52bd52d68f7413ab8f09fc47047b552f13f03e72b365d7d3d WatchSource:0}: Error finding container 569d016ff428f7a52bd52d68f7413ab8f09fc47047b552f13f03e72b365d7d3d: Status 404 returned error can't find the container with id 569d016ff428f7a52bd52d68f7413ab8f09fc47047b552f13f03e72b365d7d3d Nov 02 08:37:53.818668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:53.818634 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c41bdc-daa2-4024-8042-cb6810f56988" path="/var/lib/kubelet/pods/c9c41bdc-daa2-4024-8042-cb6810f56988/volumes" Nov 02 08:37:54.022623 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:54.022539 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:54.306704 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:54.306611 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" event={"ID":"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567","Type":"ContainerStarted","Data":"f82922b6075734289bd61c04a914e97e141d421c95970a04d4ae8164aefb1d84"} Nov 02 08:37:54.306704 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:54.306675 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" event={"ID":"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567","Type":"ContainerStarted","Data":"569d016ff428f7a52bd52d68f7413ab8f09fc47047b552f13f03e72b365d7d3d"} Nov 02 08:37:54.348257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:54.348193 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" podStartSLOduration=1.348175576 podStartE2EDuration="1.348175576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:37:54.347379956 +0000 UTC m=+4374.645660796" watchObservedRunningTime="2026-11-02 08:37:54.348175576 +0000 UTC m=+4374.646456399" Nov 02 08:37:54.918753 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:54.918665 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:55.049779 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:55.049747 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:55.314609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:55.314575 5899 generic.go:356] "Generic (PLEG): container finished" podID="9420fdfa-e0b9-4a66-bbf6-1dbf986ad567" containerID="f82922b6075734289bd61c04a914e97e141d421c95970a04d4ae8164aefb1d84" exitCode=0 Nov 02 08:37:55.314771 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:55.314665 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" event={"ID":"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567","Type":"ContainerDied","Data":"f82922b6075734289bd61c04a914e97e141d421c95970a04d4ae8164aefb1d84"} Nov 02 08:37:56.466711 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:56.466681 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" Nov 02 08:37:56.654367 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:56.654291 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-xks-ca\") pod \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\" (UID: \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\") " Nov 02 08:37:56.654367 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:56.654364 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-openshift-ca\") pod \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\" (UID: \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\") " Nov 02 08:37:56.654550 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:56.654431 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-kube-api-access-p6bf9\" (UniqueName: \"kubernetes.io/projected/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-kube-api-access-p6bf9\") pod \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\" (UID: \"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567\") " Nov 02 08:37:56.654736 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:56.654701 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-openshift-ca" pod "9420fdfa-e0b9-4a66-bbf6-1dbf986ad567" (UID: "9420fdfa-e0b9-4a66-bbf6-1dbf986ad567"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:37:56.656457 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:56.656429 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-xks-ca" pod "9420fdfa-e0b9-4a66-bbf6-1dbf986ad567" (UID: "9420fdfa-e0b9-4a66-bbf6-1dbf986ad567"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:37:56.656558 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:56.656502 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-kube-api-access-p6bf9" pod "9420fdfa-e0b9-4a66-bbf6-1dbf986ad567" (UID: "9420fdfa-e0b9-4a66-bbf6-1dbf986ad567"). InnerVolumeSpecName "kube-api-access-p6bf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:37:56.755873 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:56.755841 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6bf9\" (UniqueName: \"kubernetes.io/projected/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-kube-api-access-p6bf9\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:56.755873 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:56.755872 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:56.756058 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:56.755881 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:37:57.328339 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:57.328300 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" Nov 02 08:37:57.328493 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:57.328335 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb" event={"ID":"9420fdfa-e0b9-4a66-bbf6-1dbf986ad567","Type":"ContainerDied","Data":"569d016ff428f7a52bd52d68f7413ab8f09fc47047b552f13f03e72b365d7d3d"} Nov 02 08:37:57.328493 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:57.328415 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="569d016ff428f7a52bd52d68f7413ab8f09fc47047b552f13f03e72b365d7d3d" Nov 02 08:37:58.180419 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:58.180380 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm"] Nov 02 08:37:58.180849 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:58.180568 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" podUID="d8af1193-18d1-42b6-b887-e96f00615754" containerName="istio-proxy" containerID="cri-o://b7147a38e93cf9768ec3d9a3921852878a215e1480e8d36ec7a7c971dd065033" gracePeriod=30 Nov 02 08:37:58.277769 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:58.277737 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:37:58.415450 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:37:58.415419 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:00.017650 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:00.017614 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:00.950872 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:00.950830 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerName="main" containerID="cri-o://ceaf2f4719b7654a3dd387f2fb11a5d1e9a65653bd35e95a8f67f49fcb89d2b2" gracePeriod=45 Nov 02 08:38:00.963554 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:00.963526 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" containerID="cri-o://ecc91f62c8cbdc066e473317d4ade274246ef7a18010283402b1b37848c771d8" gracePeriod=45 Nov 02 08:38:01.194086 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.194047 5899 scope.go:122] "RemoveContainer" containerID="cbd930a5c62798d634223a6c7b9a073bf91a1435c521093824eadd273ab3f0d5" Nov 02 08:38:01.232020 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.231942 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 08:38:01.259325 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.259304 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 08:38:01.286976 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.286945 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-home\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-home\") pod \"de594282-67a5-4dc7-8170-53abd0c34041\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " Nov 02 08:38:01.287793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.286981 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-dshm\") pod \"de594282-67a5-4dc7-8170-53abd0c34041\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " Nov 02 08:38:01.287793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.287032 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6a95fe-287c-4039-9165-6464f1a757e0-kube-api-access-zw8ld\" (UniqueName: \"kubernetes.io/projected/3f6a95fe-287c-4039-9165-6464f1a757e0-kube-api-access-zw8ld\") pod \"3f6a95fe-287c-4039-9165-6464f1a757e0\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " Nov 02 08:38:01.287793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.287063 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-tmp-dir\") pod \"3f6a95fe-287c-4039-9165-6464f1a757e0\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " Nov 02 08:38:01.287793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.287090 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-dshm\") pod \"3f6a95fe-287c-4039-9165-6464f1a757e0\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " Nov 02 08:38:01.287793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.287139 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-model-cache\") pod \"3f6a95fe-287c-4039-9165-6464f1a757e0\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " Nov 02 08:38:01.287793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.287217 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-home\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-home\") pod \"3f6a95fe-287c-4039-9165-6464f1a757e0\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " Nov 02 08:38:01.287793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.287269 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-model-cache\") pod \"de594282-67a5-4dc7-8170-53abd0c34041\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " Nov 02 08:38:01.287793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.287303 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de594282-67a5-4dc7-8170-53abd0c34041-tls-certs\" (UniqueName: \"kubernetes.io/secret/de594282-67a5-4dc7-8170-53abd0c34041-tls-certs\") pod \"de594282-67a5-4dc7-8170-53abd0c34041\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " Nov 02 08:38:01.287793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.287329 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-home" pod "de594282-67a5-4dc7-8170-53abd0c34041" (UID: "de594282-67a5-4dc7-8170-53abd0c34041"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:01.287793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.287355 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de594282-67a5-4dc7-8170-53abd0c34041-kube-api-access-2pgn2\" (UniqueName: \"kubernetes.io/projected/de594282-67a5-4dc7-8170-53abd0c34041-kube-api-access-2pgn2\") pod \"de594282-67a5-4dc7-8170-53abd0c34041\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " Nov 02 08:38:01.287793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.287387 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f6a95fe-287c-4039-9165-6464f1a757e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6a95fe-287c-4039-9165-6464f1a757e0-tls-certs\") pod \"3f6a95fe-287c-4039-9165-6464f1a757e0\" (UID: \"3f6a95fe-287c-4039-9165-6464f1a757e0\") " Nov 02 08:38:01.287793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.287437 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-tmp-dir\") pod \"de594282-67a5-4dc7-8170-53abd0c34041\" (UID: \"de594282-67a5-4dc7-8170-53abd0c34041\") " Nov 02 08:38:01.288643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.287809 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:01.288643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.288007 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-tmp-dir" pod "de594282-67a5-4dc7-8170-53abd0c34041" (UID: "de594282-67a5-4dc7-8170-53abd0c34041"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:01.288643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.288156 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-model-cache" pod "3f6a95fe-287c-4039-9165-6464f1a757e0" (UID: "3f6a95fe-287c-4039-9165-6464f1a757e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:01.288643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.288337 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-home" pod "3f6a95fe-287c-4039-9165-6464f1a757e0" (UID: "3f6a95fe-287c-4039-9165-6464f1a757e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:01.288643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.288512 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-model-cache" pod "de594282-67a5-4dc7-8170-53abd0c34041" (UID: "de594282-67a5-4dc7-8170-53abd0c34041"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:01.291563 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.289271 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-tmp-dir" pod "3f6a95fe-287c-4039-9165-6464f1a757e0" (UID: "3f6a95fe-287c-4039-9165-6464f1a757e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:01.291563 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.291431 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-dshm" pod "3f6a95fe-287c-4039-9165-6464f1a757e0" (UID: "3f6a95fe-287c-4039-9165-6464f1a757e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:01.292134 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.292107 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de594282-67a5-4dc7-8170-53abd0c34041-kube-api-access-2pgn2" pod "de594282-67a5-4dc7-8170-53abd0c34041" (UID: "de594282-67a5-4dc7-8170-53abd0c34041"). InnerVolumeSpecName "kube-api-access-2pgn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:01.292917 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.292888 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-dshm" pod "de594282-67a5-4dc7-8170-53abd0c34041" (UID: "de594282-67a5-4dc7-8170-53abd0c34041"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:01.293008 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.292909 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de594282-67a5-4dc7-8170-53abd0c34041-tls-certs" pod "de594282-67a5-4dc7-8170-53abd0c34041" (UID: "de594282-67a5-4dc7-8170-53abd0c34041"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:38:01.293445 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.293413 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6a95fe-287c-4039-9165-6464f1a757e0-kube-api-access-zw8ld" pod "3f6a95fe-287c-4039-9165-6464f1a757e0" (UID: "3f6a95fe-287c-4039-9165-6464f1a757e0"). InnerVolumeSpecName "kube-api-access-zw8ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:01.293813 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.293789 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f6a95fe-287c-4039-9165-6464f1a757e0-tls-certs" pod "3f6a95fe-287c-4039-9165-6464f1a757e0" (UID: "3f6a95fe-287c-4039-9165-6464f1a757e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:38:01.350879 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.350839 5899 generic.go:356] "Generic (PLEG): container finished" podID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerID="ceaf2f4719b7654a3dd387f2fb11a5d1e9a65653bd35e95a8f67f49fcb89d2b2" exitCode=0 Nov 02 08:38:01.350879 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.350906 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" Nov 02 08:38:01.350879 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.350924 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" event={"ID":"3f6a95fe-287c-4039-9165-6464f1a757e0","Type":"ContainerDied","Data":"ceaf2f4719b7654a3dd387f2fb11a5d1e9a65653bd35e95a8f67f49fcb89d2b2"} Nov 02 08:38:01.350879 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.350974 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" event={"ID":"3f6a95fe-287c-4039-9165-6464f1a757e0","Type":"ContainerDied","Data":"2cae60f7c9c99ac954c6e1369e1f2abf64458de9900c0ee2e6afd0993736157c"} Nov 02 08:38:01.350879 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.351001 5899 scope.go:122] "RemoveContainer" containerID="ceaf2f4719b7654a3dd387f2fb11a5d1e9a65653bd35e95a8f67f49fcb89d2b2" Nov 02 08:38:01.352913 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.352765 5899 generic.go:356] "Generic (PLEG): container finished" podID="de594282-67a5-4dc7-8170-53abd0c34041" containerID="ecc91f62c8cbdc066e473317d4ade274246ef7a18010283402b1b37848c771d8" exitCode=0 Nov 02 08:38:01.352913 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.352820 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" Nov 02 08:38:01.352913 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.352850 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" event={"ID":"de594282-67a5-4dc7-8170-53abd0c34041","Type":"ContainerDied","Data":"ecc91f62c8cbdc066e473317d4ade274246ef7a18010283402b1b37848c771d8"} Nov 02 08:38:01.352913 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.352876 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k" event={"ID":"de594282-67a5-4dc7-8170-53abd0c34041","Type":"ContainerDied","Data":"573e2a67828809337f2a08972f5a83eadaf47ca9a3e4ec748ffe6cf1964e7dca"} Nov 02 08:38:01.364035 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.363742 5899 scope.go:122] "RemoveContainer" containerID="ceaf2f4719b7654a3dd387f2fb11a5d1e9a65653bd35e95a8f67f49fcb89d2b2" Nov 02 08:38:01.364432 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:38:01.364284 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceaf2f4719b7654a3dd387f2fb11a5d1e9a65653bd35e95a8f67f49fcb89d2b2\": container with ID starting with ceaf2f4719b7654a3dd387f2fb11a5d1e9a65653bd35e95a8f67f49fcb89d2b2 not found: ID does not exist" containerID="ceaf2f4719b7654a3dd387f2fb11a5d1e9a65653bd35e95a8f67f49fcb89d2b2" Nov 02 08:38:01.364432 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.364324 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceaf2f4719b7654a3dd387f2fb11a5d1e9a65653bd35e95a8f67f49fcb89d2b2"} err="failed to get container status \"ceaf2f4719b7654a3dd387f2fb11a5d1e9a65653bd35e95a8f67f49fcb89d2b2\": rpc error: code = NotFound desc = could not find container \"ceaf2f4719b7654a3dd387f2fb11a5d1e9a65653bd35e95a8f67f49fcb89d2b2\": container with ID starting with ceaf2f4719b7654a3dd387f2fb11a5d1e9a65653bd35e95a8f67f49fcb89d2b2 not found: ID does not exist" Nov 02 08:38:01.364432 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.364349 5899 scope.go:122] "RemoveContainer" containerID="ecc91f62c8cbdc066e473317d4ade274246ef7a18010283402b1b37848c771d8" Nov 02 08:38:01.375529 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.375509 5899 scope.go:122] "RemoveContainer" containerID="ecc91f62c8cbdc066e473317d4ade274246ef7a18010283402b1b37848c771d8" Nov 02 08:38:01.375842 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:38:01.375820 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc91f62c8cbdc066e473317d4ade274246ef7a18010283402b1b37848c771d8\": container with ID starting with ecc91f62c8cbdc066e473317d4ade274246ef7a18010283402b1b37848c771d8 not found: ID does not exist" containerID="ecc91f62c8cbdc066e473317d4ade274246ef7a18010283402b1b37848c771d8" Nov 02 08:38:01.375907 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.375850 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc91f62c8cbdc066e473317d4ade274246ef7a18010283402b1b37848c771d8"} err="failed to get container status \"ecc91f62c8cbdc066e473317d4ade274246ef7a18010283402b1b37848c771d8\": rpc error: code = NotFound desc = could not find container \"ecc91f62c8cbdc066e473317d4ade274246ef7a18010283402b1b37848c771d8\": container with ID starting with ecc91f62c8cbdc066e473317d4ade274246ef7a18010283402b1b37848c771d8 not found: ID does not exist" Nov 02 08:38:01.388501 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.388468 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pgn2\" (UniqueName: \"kubernetes.io/projected/de594282-67a5-4dc7-8170-53abd0c34041-kube-api-access-2pgn2\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:01.388501 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.388501 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6a95fe-287c-4039-9165-6464f1a757e0-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:01.388685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.388516 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:01.388685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.388530 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:01.388685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.388545 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw8ld\" (UniqueName: \"kubernetes.io/projected/3f6a95fe-287c-4039-9165-6464f1a757e0-kube-api-access-zw8ld\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:01.388685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.388559 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:01.388685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.388573 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:01.388685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.388585 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:01.388685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.388597 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6a95fe-287c-4039-9165-6464f1a757e0-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:01.388685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.388609 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de594282-67a5-4dc7-8170-53abd0c34041-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:01.388685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.388621 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de594282-67a5-4dc7-8170-53abd0c34041-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:01.405514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.405479 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k"] Nov 02 08:38:01.410946 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.410917 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-337764/premium-w2-337764-kserve-bc9d49c4-p5m7k"] Nov 02 08:38:01.454577 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.453678 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw"] Nov 02 08:38:01.457263 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.457231 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw"] Nov 02 08:38:01.819076 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.819037 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" path="/var/lib/kubelet/pods/3f6a95fe-287c-4039-9165-6464f1a757e0/volumes" Nov 02 08:38:01.819507 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:01.819491 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de594282-67a5-4dc7-8170-53abd0c34041" path="/var/lib/kubelet/pods/de594282-67a5-4dc7-8170-53abd0c34041/volumes" Nov 02 08:38:02.160357 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:02.160266 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-2-w2-337764-kserve-57f5c455f4-9cjfw" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 02 08:38:03.366551 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.366513 5899 generic.go:356] "Generic (PLEG): container finished" podID="d8af1193-18d1-42b6-b887-e96f00615754" containerID="b7147a38e93cf9768ec3d9a3921852878a215e1480e8d36ec7a7c971dd065033" exitCode=0 Nov 02 08:38:03.366934 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.366583 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" event={"ID":"d8af1193-18d1-42b6-b887-e96f00615754","Type":"ContainerDied","Data":"b7147a38e93cf9768ec3d9a3921852878a215e1480e8d36ec7a7c971dd065033"} Nov 02 08:38:03.439035 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.439002 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:38:03.507314 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.507275 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8af1193-18d1-42b6-b887-e96f00615754-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8af1193-18d1-42b6-b887-e96f00615754-istiod-ca-cert\") pod \"d8af1193-18d1-42b6-b887-e96f00615754\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " Nov 02 08:38:03.507530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.507358 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-socket\") pod \"d8af1193-18d1-42b6-b887-e96f00615754\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " Nov 02 08:38:03.507530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.507403 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-envoy\") pod \"d8af1193-18d1-42b6-b887-e96f00615754\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " Nov 02 08:38:03.507530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.507463 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-credential-socket\") pod \"d8af1193-18d1-42b6-b887-e96f00615754\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " Nov 02 08:38:03.507733 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.507565 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8af1193-18d1-42b6-b887-e96f00615754-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8af1193-18d1-42b6-b887-e96f00615754-istio-podinfo\") pod \"d8af1193-18d1-42b6-b887-e96f00615754\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " Nov 02 08:38:03.507733 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.507622 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-istio-token\" (UniqueName: \"kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-istio-token\") pod \"d8af1193-18d1-42b6-b887-e96f00615754\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " Nov 02 08:38:03.507733 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.507688 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-data\") pod \"d8af1193-18d1-42b6-b887-e96f00615754\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " Nov 02 08:38:03.507733 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.507723 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-certs\") pod \"d8af1193-18d1-42b6-b887-e96f00615754\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " Nov 02 08:38:03.507970 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.507753 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-kube-api-access-qf7cl\" (UniqueName: \"kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-kube-api-access-qf7cl\") pod \"d8af1193-18d1-42b6-b887-e96f00615754\" (UID: \"d8af1193-18d1-42b6-b887-e96f00615754\") " Nov 02 08:38:03.507970 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.507794 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8af1193-18d1-42b6-b887-e96f00615754-istiod-ca-cert" pod "d8af1193-18d1-42b6-b887-e96f00615754" (UID: "d8af1193-18d1-42b6-b887-e96f00615754"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:38:03.508464 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.508119 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8af1193-18d1-42b6-b887-e96f00615754-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:03.509141 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.508873 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-certs" pod "d8af1193-18d1-42b6-b887-e96f00615754" (UID: "d8af1193-18d1-42b6-b887-e96f00615754"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:03.509407 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.509382 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-data" pod "d8af1193-18d1-42b6-b887-e96f00615754" (UID: "d8af1193-18d1-42b6-b887-e96f00615754"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:03.509716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.509697 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-socket" pod "d8af1193-18d1-42b6-b887-e96f00615754" (UID: "d8af1193-18d1-42b6-b887-e96f00615754"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:03.509890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.509862 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-credential-socket" pod "d8af1193-18d1-42b6-b887-e96f00615754" (UID: "d8af1193-18d1-42b6-b887-e96f00615754"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:03.510819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.510789 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-kube-api-access-qf7cl" pod "d8af1193-18d1-42b6-b887-e96f00615754" (UID: "d8af1193-18d1-42b6-b887-e96f00615754"). InnerVolumeSpecName "kube-api-access-qf7cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:03.512268 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.512108 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-istio-token" pod "d8af1193-18d1-42b6-b887-e96f00615754" (UID: "d8af1193-18d1-42b6-b887-e96f00615754"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:03.512268 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.512215 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8af1193-18d1-42b6-b887-e96f00615754-istio-podinfo" pod "d8af1193-18d1-42b6-b887-e96f00615754" (UID: "d8af1193-18d1-42b6-b887-e96f00615754"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:38:03.514958 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.514931 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:03.518603 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.518576 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-envoy" pod "d8af1193-18d1-42b6-b887-e96f00615754" (UID: "d8af1193-18d1-42b6-b887-e96f00615754"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:03.609352 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.609313 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:03.609352 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.609346 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:03.609352 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.609355 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:03.609601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.609364 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf7cl\" (UniqueName: \"kubernetes.io/projected/d8af1193-18d1-42b6-b887-e96f00615754-kube-api-access-qf7cl\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:03.609601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.609374 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:03.609601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.609382 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:03.609601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.609390 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8af1193-18d1-42b6-b887-e96f00615754-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:03.609601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.609397 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8af1193-18d1-42b6-b887-e96f00615754-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:03.652998 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:03.652948 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:04.373552 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:04.373520 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" Nov 02 08:38:04.373552 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:04.373546 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm" event={"ID":"d8af1193-18d1-42b6-b887-e96f00615754","Type":"ContainerDied","Data":"4e865e80cc4e615633b4e001fddae244b871081028cd76d474f246396e3893e3"} Nov 02 08:38:04.374137 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:04.373613 5899 scope.go:122] "RemoveContainer" containerID="b7147a38e93cf9768ec3d9a3921852878a215e1480e8d36ec7a7c971dd065033" Nov 02 08:38:04.440651 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:04.440612 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm"] Nov 02 08:38:04.453807 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:04.453778 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2tbm"] Nov 02 08:38:04.931788 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:04.931746 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:05.331624 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:05.331593 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:05.818359 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:05.818322 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8af1193-18d1-42b6-b887-e96f00615754" path="/var/lib/kubelet/pods/d8af1193-18d1-42b6-b887-e96f00615754/volumes" Nov 02 08:38:06.623824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:06.623787 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:06.778433 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:06.778395 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:06.918239 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:06.918121 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:06.993617 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:06.993579 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p"] Nov 02 08:38:06.993811 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:06.993788 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" podUID="bc8490b8-4f35-42eb-9140-6e0bbc276f93" containerName="istio-proxy" containerID="cri-o://49bb748d6f9dca01327695d67035c7df53813f0cddc8d7e5226899744cb38d5d" gracePeriod=30 Nov 02 08:38:07.080385 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:07.080345 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:10.324475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:10.324439 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:11.032194 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:11.032143 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:11.232796 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:11.232756 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:12.253787 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.253761 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 08:38:12.289540 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.289457 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-podinfo\") pod \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " Nov 02 08:38:12.289540 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.289498 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-kube-api-access-47s5v\" (UniqueName: \"kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-kube-api-access-47s5v\") pod \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " Nov 02 08:38:12.289746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.289562 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-certs\") pod \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " Nov 02 08:38:12.289746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.289591 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-token\" (UniqueName: \"kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-token\") pod \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " Nov 02 08:38:12.289746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.289624 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-data\") pod \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " Nov 02 08:38:12.289746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.289684 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-socket\") pod \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " Nov 02 08:38:12.289746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.289721 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-envoy\") pod \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " Nov 02 08:38:12.289982 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.289782 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istiod-ca-cert\") pod \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " Nov 02 08:38:12.289982 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.289833 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-credential-socket\") pod \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\" (UID: \"bc8490b8-4f35-42eb-9140-6e0bbc276f93\") " Nov 02 08:38:12.290104 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.289997 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-certs" pod "bc8490b8-4f35-42eb-9140-6e0bbc276f93" (UID: "bc8490b8-4f35-42eb-9140-6e0bbc276f93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:12.290104 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.290007 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-data" pod "bc8490b8-4f35-42eb-9140-6e0bbc276f93" (UID: "bc8490b8-4f35-42eb-9140-6e0bbc276f93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:12.290322 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.290280 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-credential-socket" pod "bc8490b8-4f35-42eb-9140-6e0bbc276f93" (UID: "bc8490b8-4f35-42eb-9140-6e0bbc276f93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:12.290487 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.290365 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:12.290487 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.290385 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:12.290487 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.290398 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:12.290487 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.290408 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-socket" pod "bc8490b8-4f35-42eb-9140-6e0bbc276f93" (UID: "bc8490b8-4f35-42eb-9140-6e0bbc276f93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:12.290487 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.290428 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istiod-ca-cert" pod "bc8490b8-4f35-42eb-9140-6e0bbc276f93" (UID: "bc8490b8-4f35-42eb-9140-6e0bbc276f93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:38:12.291890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.291863 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-token" pod "bc8490b8-4f35-42eb-9140-6e0bbc276f93" (UID: "bc8490b8-4f35-42eb-9140-6e0bbc276f93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:12.291890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.291881 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-podinfo" pod "bc8490b8-4f35-42eb-9140-6e0bbc276f93" (UID: "bc8490b8-4f35-42eb-9140-6e0bbc276f93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:38:12.292053 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.291968 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-kube-api-access-47s5v" pod "bc8490b8-4f35-42eb-9140-6e0bbc276f93" (UID: "bc8490b8-4f35-42eb-9140-6e0bbc276f93"). InnerVolumeSpecName "kube-api-access-47s5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:12.292053 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.291967 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-envoy" pod "bc8490b8-4f35-42eb-9140-6e0bbc276f93" (UID: "bc8490b8-4f35-42eb-9140-6e0bbc276f93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:12.391135 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.391097 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:12.391135 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.391128 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:12.391135 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.391139 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:12.391429 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.391147 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:12.391429 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.391155 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8490b8-4f35-42eb-9140-6e0bbc276f93-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:12.391429 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.391176 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47s5v\" (UniqueName: \"kubernetes.io/projected/bc8490b8-4f35-42eb-9140-6e0bbc276f93-kube-api-access-47s5v\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:12.420851 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.420812 5899 generic.go:356] "Generic (PLEG): container finished" podID="bc8490b8-4f35-42eb-9140-6e0bbc276f93" containerID="49bb748d6f9dca01327695d67035c7df53813f0cddc8d7e5226899744cb38d5d" exitCode=0 Nov 02 08:38:12.421060 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.420864 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" Nov 02 08:38:12.421060 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.420893 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" event={"ID":"bc8490b8-4f35-42eb-9140-6e0bbc276f93","Type":"ContainerDied","Data":"49bb748d6f9dca01327695d67035c7df53813f0cddc8d7e5226899744cb38d5d"} Nov 02 08:38:12.421060 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.420964 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p" event={"ID":"bc8490b8-4f35-42eb-9140-6e0bbc276f93","Type":"ContainerDied","Data":"e7c512464f66a3cb9d72ff699acef99f4503cb1d510ab2af295822eba46cc574"} Nov 02 08:38:12.421060 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.420994 5899 scope.go:122] "RemoveContainer" containerID="49bb748d6f9dca01327695d67035c7df53813f0cddc8d7e5226899744cb38d5d" Nov 02 08:38:12.430549 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.430526 5899 scope.go:122] "RemoveContainer" containerID="49bb748d6f9dca01327695d67035c7df53813f0cddc8d7e5226899744cb38d5d" Nov 02 08:38:12.430840 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:38:12.430821 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49bb748d6f9dca01327695d67035c7df53813f0cddc8d7e5226899744cb38d5d\": container with ID starting with 49bb748d6f9dca01327695d67035c7df53813f0cddc8d7e5226899744cb38d5d not found: ID does not exist" containerID="49bb748d6f9dca01327695d67035c7df53813f0cddc8d7e5226899744cb38d5d" Nov 02 08:38:12.430897 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.430848 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49bb748d6f9dca01327695d67035c7df53813f0cddc8d7e5226899744cb38d5d"} err="failed to get container status \"49bb748d6f9dca01327695d67035c7df53813f0cddc8d7e5226899744cb38d5d\": rpc error: code = NotFound desc = could not find container \"49bb748d6f9dca01327695d67035c7df53813f0cddc8d7e5226899744cb38d5d\": container with ID starting with 49bb748d6f9dca01327695d67035c7df53813f0cddc8d7e5226899744cb38d5d not found: ID does not exist" Nov 02 08:38:12.475172 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.475123 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p"] Nov 02 08:38:12.479625 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:12.479591 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-nr52p"] Nov 02 08:38:13.141118 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:38:13.141079 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:38:13.819483 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:13.819451 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc8490b8-4f35-42eb-9140-6e0bbc276f93" path="/var/lib/kubelet/pods/bc8490b8-4f35-42eb-9140-6e0bbc276f93/volumes" Nov 02 08:38:14.029067 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:14.029030 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:14.164328 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:14.164196 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:23.415562 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:23.415519 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:23.545974 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:23.545933 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:24.155031 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:24.154987 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:24.877082 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:24.877047 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:25.182903 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:25.182801 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6"] Nov 02 08:38:25.250629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:25.250596 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:25.485803 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:25.485765 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" podUID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" containerName="istio-proxy" containerID="cri-o://fbb38da793d95bcb5ebfddbd2a446afa0d328e3020ee29f4dae1d88d5170936b" gracePeriod=30 Nov 02 08:38:26.873221 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:26.873138 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xjq52_b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae/kuadrant-console-plugin/0.log" Nov 02 08:38:27.099623 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:27.099585 5899 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 08:38:27.099792 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:27.099641 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" podUID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 08:38:28.885428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:28.885394 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-g8nbx_5ac39273-9863-4a60-86f4-29e64cbe2946/manager/0.log" Nov 02 08:38:29.082058 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:29.082024 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kv7wp_62de7c44-3318-4553-b3d0-f0124ed28ac4/manager/0.log" Nov 02 08:38:29.284243 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:29.284197 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-kqv85_580bc81e-1b83-472a-9af3-153ba348eab7/manager/0.log" Nov 02 08:38:29.823279 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:29.823244 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:29.948685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:29.948658 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pq62z_787dc60b-c7be-4b20-9b11-daf3140a7589/server/0.log" Nov 02 08:38:29.963813 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:29.963783 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:30.744510 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.744478 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 08:38:30.873987 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.873896 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-socket\") pod \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " Nov 02 08:38:30.873987 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.873948 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-token\" (UniqueName: \"kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-token\") pod \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " Nov 02 08:38:30.873987 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.873983 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-certs\") pod \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " Nov 02 08:38:30.874300 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874008 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-kube-api-access-glhjs\" (UniqueName: \"kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-kube-api-access-glhjs\") pod \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " Nov 02 08:38:30.874300 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874030 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-podinfo\") pod \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " Nov 02 08:38:30.874300 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874062 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-data\") pod \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " Nov 02 08:38:30.874300 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874120 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-envoy\") pod \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " Nov 02 08:38:30.874300 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874152 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-credential-socket\") pod \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " Nov 02 08:38:30.874300 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874222 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istiod-ca-cert\") pod \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\" (UID: \"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e\") " Nov 02 08:38:30.874300 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874235 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-socket" pod "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" (UID: "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:30.874655 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874361 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-certs" pod "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" (UID: "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:30.874655 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874501 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-credential-socket" pod "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" (UID: "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:30.874655 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874519 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-data" pod "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" (UID: "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:30.874824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874696 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istiod-ca-cert" pod "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" (UID: "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:38:30.874885 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874848 5899 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-data\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:30.874885 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874871 5899 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-credential-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:30.875013 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874886 5899 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istiod-ca-cert\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:30.875013 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874900 5899 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-socket\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:30.875013 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.874913 5899 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-workload-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:30.876531 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.876504 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-envoy" pod "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" (UID: "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:38:30.876651 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.876612 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-kube-api-access-glhjs" pod "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" (UID: "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e"). InnerVolumeSpecName "kube-api-access-glhjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:30.876713 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.876676 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-token" pod "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" (UID: "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:30.876759 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.876744 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-podinfo" pod "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" (UID: "fd0427a3-509d-4e27-8cf2-e5782fd4ff0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 08:38:30.976426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.976389 5899 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-token\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:30.976426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.976419 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhjs\" (UniqueName: \"kubernetes.io/projected/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-kube-api-access-glhjs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:30.976426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.976428 5899 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-podinfo\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:30.976426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:30.976437 5899 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e-istio-envoy\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:31.517016 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:31.516979 5899 generic.go:356] "Generic (PLEG): container finished" podID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" containerID="fbb38da793d95bcb5ebfddbd2a446afa0d328e3020ee29f4dae1d88d5170936b" exitCode=0 Nov 02 08:38:31.517229 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:31.517021 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" event={"ID":"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e","Type":"ContainerDied","Data":"fbb38da793d95bcb5ebfddbd2a446afa0d328e3020ee29f4dae1d88d5170936b"} Nov 02 08:38:31.517229 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:31.517029 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" Nov 02 08:38:31.517229 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:31.517067 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6" event={"ID":"fd0427a3-509d-4e27-8cf2-e5782fd4ff0e","Type":"ContainerDied","Data":"4e763662b888a6a2fd4e9c24bafa4d764b0ef0544a0366be6cfb845d3141b411"} Nov 02 08:38:31.517229 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:31.517089 5899 scope.go:122] "RemoveContainer" containerID="fbb38da793d95bcb5ebfddbd2a446afa0d328e3020ee29f4dae1d88d5170936b" Nov 02 08:38:31.536790 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:31.536765 5899 scope.go:122] "RemoveContainer" containerID="fbb38da793d95bcb5ebfddbd2a446afa0d328e3020ee29f4dae1d88d5170936b" Nov 02 08:38:31.537090 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:38:31.537068 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb38da793d95bcb5ebfddbd2a446afa0d328e3020ee29f4dae1d88d5170936b\": container with ID starting with fbb38da793d95bcb5ebfddbd2a446afa0d328e3020ee29f4dae1d88d5170936b not found: ID does not exist" containerID="fbb38da793d95bcb5ebfddbd2a446afa0d328e3020ee29f4dae1d88d5170936b" Nov 02 08:38:31.537142 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:31.537099 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb38da793d95bcb5ebfddbd2a446afa0d328e3020ee29f4dae1d88d5170936b"} err="failed to get container status \"fbb38da793d95bcb5ebfddbd2a446afa0d328e3020ee29f4dae1d88d5170936b\": rpc error: code = NotFound desc = could not find container \"fbb38da793d95bcb5ebfddbd2a446afa0d328e3020ee29f4dae1d88d5170936b\": container with ID starting with fbb38da793d95bcb5ebfddbd2a446afa0d328e3020ee29f4dae1d88d5170936b not found: ID does not exist" Nov 02 08:38:31.585505 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:31.585474 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6"] Nov 02 08:38:31.598736 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:31.598704 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-b9wp6"] Nov 02 08:38:31.819304 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:31.819228 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" path="/var/lib/kubelet/pods/fd0427a3-509d-4e27-8cf2-e5782fd4ff0e/volumes" Nov 02 08:38:34.971064 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:34.971023 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:35.484185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:35.473969 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:35.610550 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:35.610517 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:38.010175 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:38.010123 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:38.286617 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:38.286525 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:39.798513 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:39.798478 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:39.951899 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:39.951861 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:41.670913 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:41.670816 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:43.140575 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:38:43.140542 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:38:43.514715 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:43.514653 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:45.345382 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:45.345347 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:47.262221 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:47.262180 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:49.144513 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:49.144474 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:51.015518 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:51.015477 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:52.713329 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:52.713293 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:54.636476 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.636442 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:54.933392 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933304 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x"] Nov 02 08:38:54.933710 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933695 5899 cpu_manager.go:414] "containerMap: removing container" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerName="main" Nov 02 08:38:54.933710 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933710 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerName="main" Nov 02 08:38:54.933832 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933724 5899 cpu_manager.go:414] "containerMap: removing container" podUID="d8af1193-18d1-42b6-b887-e96f00615754" containerName="istio-proxy" Nov 02 08:38:54.933832 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933729 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8af1193-18d1-42b6-b887-e96f00615754" containerName="istio-proxy" Nov 02 08:38:54.933832 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933738 5899 cpu_manager.go:414] "containerMap: removing container" podUID="bc8490b8-4f35-42eb-9140-6e0bbc276f93" containerName="istio-proxy" Nov 02 08:38:54.933832 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933743 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8490b8-4f35-42eb-9140-6e0bbc276f93" containerName="istio-proxy" Nov 02 08:38:54.933832 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933750 5899 cpu_manager.go:414] "containerMap: removing container" podUID="9420fdfa-e0b9-4a66-bbf6-1dbf986ad567" containerName="revoke-keys" Nov 02 08:38:54.933832 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933754 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9420fdfa-e0b9-4a66-bbf6-1dbf986ad567" containerName="revoke-keys" Nov 02 08:38:54.933832 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933772 5899 cpu_manager.go:414] "containerMap: removing container" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" Nov 02 08:38:54.933832 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933778 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" Nov 02 08:38:54.933832 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933786 5899 cpu_manager.go:414] "containerMap: removing container" podUID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" containerName="istio-proxy" Nov 02 08:38:54.933832 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933791 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" containerName="istio-proxy" Nov 02 08:38:54.934320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933857 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="de594282-67a5-4dc7-8170-53abd0c34041" containerName="main" Nov 02 08:38:54.934320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933865 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0427a3-509d-4e27-8cf2-e5782fd4ff0e" containerName="istio-proxy" Nov 02 08:38:54.934320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933877 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f6a95fe-287c-4039-9165-6464f1a757e0" containerName="main" Nov 02 08:38:54.934320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933884 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8af1193-18d1-42b6-b887-e96f00615754" containerName="istio-proxy" Nov 02 08:38:54.934320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933889 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8490b8-4f35-42eb-9140-6e0bbc276f93" containerName="istio-proxy" Nov 02 08:38:54.934320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.933894 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="9420fdfa-e0b9-4a66-bbf6-1dbf986ad567" containerName="revoke-keys" Nov 02 08:38:54.937301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.937280 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" Nov 02 08:38:54.955523 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.955394 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:38:54.955523 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.955402 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:38:54.970004 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.969973 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x"] Nov 02 08:38:54.993498 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.993458 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-466229b7-z7k8x\" (UID: \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" Nov 02 08:38:54.993691 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.993525 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-466229b7-z7k8x\" (UID: \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" Nov 02 08:38:54.993691 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:54.993629 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kklc\" (UniqueName: \"kubernetes.io/projected/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-kube-api-access-4kklc\") pod \"maas-api-revoke-keys-e2e-ait-466229b7-z7k8x\" (UID: \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" Nov 02 08:38:55.094210 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:55.094153 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-466229b7-z7k8x\" (UID: \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" Nov 02 08:38:55.094407 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:55.094277 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kklc\" (UniqueName: \"kubernetes.io/projected/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-kube-api-access-4kklc\") pod \"maas-api-revoke-keys-e2e-ait-466229b7-z7k8x\" (UID: \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" Nov 02 08:38:55.094407 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:55.094349 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-466229b7-z7k8x\" (UID: \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" Nov 02 08:38:55.095136 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:55.095109 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-466229b7-z7k8x\" (UID: \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" Nov 02 08:38:55.096997 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:55.096972 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-466229b7-z7k8x\" (UID: \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" Nov 02 08:38:55.120147 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:55.120108 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kklc\" (UniqueName: \"kubernetes.io/projected/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-kube-api-access-4kklc\") pod \"maas-api-revoke-keys-e2e-ait-466229b7-z7k8x\" (UID: \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" Nov 02 08:38:55.252951 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:55.252918 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" Nov 02 08:38:55.551199 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:55.551154 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x"] Nov 02 08:38:55.554214 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:38:55.554186 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3334b0_bd33_4b7c_8854_5b0c5188b5b7.slice/crio-7b905d436d19da07e00f2823fb4a7e1aa2576a1238920ae750faf865e6319254 WatchSource:0}: Error finding container 7b905d436d19da07e00f2823fb4a7e1aa2576a1238920ae750faf865e6319254: Status 404 returned error can't find the container with id 7b905d436d19da07e00f2823fb4a7e1aa2576a1238920ae750faf865e6319254 Nov 02 08:38:55.635656 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:55.635618 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" event={"ID":"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7","Type":"ContainerStarted","Data":"7b905d436d19da07e00f2823fb4a7e1aa2576a1238920ae750faf865e6319254"} Nov 02 08:38:56.532749 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:56.532712 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:56.641668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:56.641573 5899 generic.go:356] "Generic (PLEG): container finished" podID="7a3334b0-bd33-4b7c-8854-5b0c5188b5b7" containerID="abde0dd204e48f355e66192b290be5cea7e4d25bf1116e1296f2896ca88a2347" exitCode=0 Nov 02 08:38:56.641827 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:56.641657 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" event={"ID":"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7","Type":"ContainerDied","Data":"abde0dd204e48f355e66192b290be5cea7e4d25bf1116e1296f2896ca88a2347"} Nov 02 08:38:57.775654 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:57.775627 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" Nov 02 08:38:57.817217 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:57.817187 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-kube-api-access-4kklc\" (UniqueName: \"kubernetes.io/projected/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-kube-api-access-4kklc\") pod \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\" (UID: \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\") " Nov 02 08:38:57.817405 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:57.817276 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-xks-ca\") pod \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\" (UID: \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\") " Nov 02 08:38:57.817405 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:57.817325 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-openshift-ca\") pod \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\" (UID: \"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7\") " Nov 02 08:38:57.817815 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:57.817770 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-openshift-ca" pod "7a3334b0-bd33-4b7c-8854-5b0c5188b5b7" (UID: "7a3334b0-bd33-4b7c-8854-5b0c5188b5b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:38:57.819734 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:57.819708 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-kube-api-access-4kklc" pod "7a3334b0-bd33-4b7c-8854-5b0c5188b5b7" (UID: "7a3334b0-bd33-4b7c-8854-5b0c5188b5b7"). InnerVolumeSpecName "kube-api-access-4kklc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:38:57.819883 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:57.819800 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-xks-ca" pod "7a3334b0-bd33-4b7c-8854-5b0c5188b5b7" (UID: "7a3334b0-bd33-4b7c-8854-5b0c5188b5b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:38:57.918271 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:57.918193 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kklc\" (UniqueName: \"kubernetes.io/projected/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-kube-api-access-4kklc\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:57.918271 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:57.918222 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:57.918271 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:57.918234 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:38:58.345784 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:58.345752 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:38:58.652359 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:58.652266 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" Nov 02 08:38:58.652359 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:58.652299 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x" event={"ID":"7a3334b0-bd33-4b7c-8854-5b0c5188b5b7","Type":"ContainerDied","Data":"7b905d436d19da07e00f2823fb4a7e1aa2576a1238920ae750faf865e6319254"} Nov 02 08:38:58.652551 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:38:58.652359 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b905d436d19da07e00f2823fb4a7e1aa2576a1238920ae750faf865e6319254" Nov 02 08:39:00.111072 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:00.111043 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:01.905772 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:01.905735 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:03.612613 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:03.612577 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:05.484946 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:05.484916 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:07.679042 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:07.679006 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:07.785481 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:07.785447 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x"] Nov 02 08:39:07.798875 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:07.798840 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-466229b7-z7k8x"] Nov 02 08:39:07.818518 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:07.818486 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3334b0-bd33-4b7c-8854-5b0c5188b5b7" path="/var/lib/kubelet/pods/7a3334b0-bd33-4b7c-8854-5b0c5188b5b7/volumes" Nov 02 08:39:07.827119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:07.827095 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:10.220858 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:10.220818 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:10.408433 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:10.408400 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:13.140824 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:39:13.140786 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:39:28.483569 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.483535 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n"] Nov 02 08:39:28.484087 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.483951 5899 cpu_manager.go:414] "containerMap: removing container" podUID="7a3334b0-bd33-4b7c-8854-5b0c5188b5b7" containerName="revoke-keys" Nov 02 08:39:28.484087 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.483963 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3334b0-bd33-4b7c-8854-5b0c5188b5b7" containerName="revoke-keys" Nov 02 08:39:28.484087 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.484049 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3334b0-bd33-4b7c-8854-5b0c5188b5b7" containerName="revoke-keys" Nov 02 08:39:28.487258 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.487236 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.518389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.518358 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n"] Nov 02 08:39:28.628760 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.628727 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vvzm\" (UniqueName: \"kubernetes.io/projected/14f48878-440e-45ac-900a-a29238f348c9-kube-api-access-9vvzm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.628760 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.628761 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f48878-440e-45ac-900a-a29238f348c9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.629034 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.628782 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.629034 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.628807 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.629034 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.628874 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f48878-440e-45ac-900a-a29238f348c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.629034 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.629017 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.629325 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.629060 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.629325 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.629204 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f48878-440e-45ac-900a-a29238f348c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.629325 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.629242 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.730415 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.730375 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.730612 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.730439 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vvzm\" (UniqueName: \"kubernetes.io/projected/14f48878-440e-45ac-900a-a29238f348c9-kube-api-access-9vvzm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.730612 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.730466 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f48878-440e-45ac-900a-a29238f348c9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.730612 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.730493 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.730612 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.730526 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.730612 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.730548 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f48878-440e-45ac-900a-a29238f348c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.730942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.730617 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.730942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.730658 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.730942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.730725 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f48878-440e-45ac-900a-a29238f348c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.730942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.730827 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.730942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.730851 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.730942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.730918 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.731267 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.731047 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.731392 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.731374 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f48878-440e-45ac-900a-a29238f348c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.732959 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.732938 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f48878-440e-45ac-900a-a29238f348c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.733301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.733280 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f48878-440e-45ac-900a-a29238f348c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.754492 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.754435 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f48878-440e-45ac-900a-a29238f348c9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.754764 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.754739 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vvzm\" (UniqueName: \"kubernetes.io/projected/14f48878-440e-45ac-900a-a29238f348c9-kube-api-access-9vvzm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hkc5n\" (UID: \"14f48878-440e-45ac-900a-a29238f348c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:28.804878 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:28.804840 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:29.123133 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:29.123047 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n"] Nov 02 08:39:29.126123 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:39:29.126091 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f48878_440e_45ac_900a_a29238f348c9.slice/crio-63fcfbc191cd901002bad4e83d2a6327d7f2b3e6851393aca81ccac95b23efd1 WatchSource:0}: Error finding container 63fcfbc191cd901002bad4e83d2a6327d7f2b3e6851393aca81ccac95b23efd1: Status 404 returned error can't find the container with id 63fcfbc191cd901002bad4e83d2a6327d7f2b3e6851393aca81ccac95b23efd1 Nov 02 08:39:29.128324 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:29.128292 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:39:29.128422 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:29.128355 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:39:29.128422 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:29.128384 5899 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 08:39:29.806522 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:29.806484 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" event={"ID":"14f48878-440e-45ac-900a-a29238f348c9","Type":"ContainerStarted","Data":"6424fff003f9ad534d798b48537bc5abaebed5dd7fa3f9beb4032d60d953679c"} Nov 02 08:39:29.806910 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:29.806543 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" event={"ID":"14f48878-440e-45ac-900a-a29238f348c9","Type":"ContainerStarted","Data":"63fcfbc191cd901002bad4e83d2a6327d7f2b3e6851393aca81ccac95b23efd1"} Nov 02 08:39:29.884153 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:29.884086 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" podStartSLOduration=1.8840666050000001 podStartE2EDuration="1.884066605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:39:29.868770605 +0000 UTC m=+4470.167051433" watchObservedRunningTime="2026-11-02 08:39:29.884066605 +0000 UTC m=+4470.182347431" Nov 02 08:39:30.806052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:30.806012 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:30.807717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:30.807690 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hkc5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:39:30.808084 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:30.807737 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" podUID="14f48878-440e-45ac-900a-a29238f348c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 02 08:39:31.806150 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:31.806114 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hkc5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:39:31.806360 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:31.806196 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" podUID="14f48878-440e-45ac-900a-a29238f348c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 02 08:39:32.736173 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:32.736135 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:32.805587 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:32.805550 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hkc5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:39:32.805772 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:32.805611 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" podUID="14f48878-440e-45ac-900a-a29238f348c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 02 08:39:32.890960 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:32.890923 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:33.805292 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:33.805227 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hkc5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:39:33.805707 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:33.805318 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" podUID="14f48878-440e-45ac-900a-a29238f348c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 02 08:39:34.805888 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:34.805848 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hkc5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:39:34.806410 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:34.805910 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" podUID="14f48878-440e-45ac-900a-a29238f348c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 02 08:39:35.248128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.248089 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm"] Nov 02 08:39:35.251941 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.251920 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" Nov 02 08:39:35.282879 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.282842 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:39:35.282879 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.282868 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:39:35.284346 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.284320 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm"] Nov 02 08:39:35.317932 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.317900 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1ca139-ad15-4981-9265-b85548218656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-glbtm\" (UID: \"6d1ca139-ad15-4981-9265-b85548218656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" Nov 02 08:39:35.318073 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.318038 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1ca139-ad15-4981-9265-b85548218656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-glbtm\" (UID: \"6d1ca139-ad15-4981-9265-b85548218656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" Nov 02 08:39:35.318073 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.318068 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drlwv\" (UniqueName: \"kubernetes.io/projected/6d1ca139-ad15-4981-9265-b85548218656-kube-api-access-drlwv\") pod \"maas-api-revoke-sub-85237e45-glbtm\" (UID: \"6d1ca139-ad15-4981-9265-b85548218656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" Nov 02 08:39:35.419078 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.419044 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1ca139-ad15-4981-9265-b85548218656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-glbtm\" (UID: \"6d1ca139-ad15-4981-9265-b85548218656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" Nov 02 08:39:35.419078 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.419082 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drlwv\" (UniqueName: \"kubernetes.io/projected/6d1ca139-ad15-4981-9265-b85548218656-kube-api-access-drlwv\") pod \"maas-api-revoke-sub-85237e45-glbtm\" (UID: \"6d1ca139-ad15-4981-9265-b85548218656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" Nov 02 08:39:35.419324 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.419130 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1ca139-ad15-4981-9265-b85548218656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-glbtm\" (UID: \"6d1ca139-ad15-4981-9265-b85548218656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" Nov 02 08:39:35.419870 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.419840 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1ca139-ad15-4981-9265-b85548218656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-glbtm\" (UID: \"6d1ca139-ad15-4981-9265-b85548218656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" Nov 02 08:39:35.421609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.421583 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1ca139-ad15-4981-9265-b85548218656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-glbtm\" (UID: \"6d1ca139-ad15-4981-9265-b85548218656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" Nov 02 08:39:35.443333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.443307 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drlwv\" (UniqueName: \"kubernetes.io/projected/6d1ca139-ad15-4981-9265-b85548218656-kube-api-access-drlwv\") pod \"maas-api-revoke-sub-85237e45-glbtm\" (UID: \"6d1ca139-ad15-4981-9265-b85548218656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" Nov 02 08:39:35.565626 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.565532 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" Nov 02 08:39:35.805594 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.805559 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hkc5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 02 08:39:35.805780 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.805613 5899 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" podUID="14f48878-440e-45ac-900a-a29238f348c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 02 08:39:35.864960 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:35.864927 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm"] Nov 02 08:39:35.867620 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:39:35.867592 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1ca139_ad15_4981_9265_b85548218656.slice/crio-fd7b0f6cf776f0205233186e1c524464d6d240ffa8e7739a536354ed453d9155 WatchSource:0}: Error finding container fd7b0f6cf776f0205233186e1c524464d6d240ffa8e7739a536354ed453d9155: Status 404 returned error can't find the container with id fd7b0f6cf776f0205233186e1c524464d6d240ffa8e7739a536354ed453d9155 Nov 02 08:39:36.842443 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:36.842407 5899 generic.go:356] "Generic (PLEG): container finished" podID="6d1ca139-ad15-4981-9265-b85548218656" containerID="85464df1f8a4a33133514c4cb6418ac0e3af4903c9dfc31ec1ab7c5554d58dbb" exitCode=0 Nov 02 08:39:36.842627 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:36.842490 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" event={"ID":"6d1ca139-ad15-4981-9265-b85548218656","Type":"ContainerDied","Data":"85464df1f8a4a33133514c4cb6418ac0e3af4903c9dfc31ec1ab7c5554d58dbb"} Nov 02 08:39:36.842627 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:36.842550 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" event={"ID":"6d1ca139-ad15-4981-9265-b85548218656","Type":"ContainerStarted","Data":"fd7b0f6cf776f0205233186e1c524464d6d240ffa8e7739a536354ed453d9155"} Nov 02 08:39:37.302424 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:37.302389 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:37.302854 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:37.302695 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:37.303428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:37.303396 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" Nov 02 08:39:37.985892 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:37.985868 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" Nov 02 08:39:38.044975 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:38.044940 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d1ca139-ad15-4981-9265-b85548218656-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1ca139-ad15-4981-9265-b85548218656-maas-api-openshift-ca\") pod \"6d1ca139-ad15-4981-9265-b85548218656\" (UID: \"6d1ca139-ad15-4981-9265-b85548218656\") " Nov 02 08:39:38.045118 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:38.045038 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d1ca139-ad15-4981-9265-b85548218656-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1ca139-ad15-4981-9265-b85548218656-maas-api-xks-ca\") pod \"6d1ca139-ad15-4981-9265-b85548218656\" (UID: \"6d1ca139-ad15-4981-9265-b85548218656\") " Nov 02 08:39:38.045202 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:38.045134 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d1ca139-ad15-4981-9265-b85548218656-kube-api-access-drlwv\" (UniqueName: \"kubernetes.io/projected/6d1ca139-ad15-4981-9265-b85548218656-kube-api-access-drlwv\") pod \"6d1ca139-ad15-4981-9265-b85548218656\" (UID: \"6d1ca139-ad15-4981-9265-b85548218656\") " Nov 02 08:39:38.045411 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:38.045375 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d1ca139-ad15-4981-9265-b85548218656-maas-api-openshift-ca" pod "6d1ca139-ad15-4981-9265-b85548218656" (UID: "6d1ca139-ad15-4981-9265-b85548218656"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:39:38.047284 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:38.047227 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1ca139-ad15-4981-9265-b85548218656-kube-api-access-drlwv" pod "6d1ca139-ad15-4981-9265-b85548218656" (UID: "6d1ca139-ad15-4981-9265-b85548218656"). InnerVolumeSpecName "kube-api-access-drlwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:39:38.047284 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:38.047234 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d1ca139-ad15-4981-9265-b85548218656-maas-api-xks-ca" pod "6d1ca139-ad15-4981-9265-b85548218656" (UID: "6d1ca139-ad15-4981-9265-b85548218656"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:39:38.145940 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:38.145852 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1ca139-ad15-4981-9265-b85548218656-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:39:38.145940 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:38.145888 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drlwv\" (UniqueName: \"kubernetes.io/projected/6d1ca139-ad15-4981-9265-b85548218656-kube-api-access-drlwv\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:39:38.145940 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:38.145903 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1ca139-ad15-4981-9265-b85548218656-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:39:38.854364 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:38.854332 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" Nov 02 08:39:38.854885 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:38.854374 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm" event={"ID":"6d1ca139-ad15-4981-9265-b85548218656","Type":"ContainerDied","Data":"fd7b0f6cf776f0205233186e1c524464d6d240ffa8e7739a536354ed453d9155"} Nov 02 08:39:38.854885 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:38.854423 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd7b0f6cf776f0205233186e1c524464d6d240ffa8e7739a536354ed453d9155" Nov 02 08:39:43.140751 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:39:43.140708 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:39:47.050594 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:47.050547 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:49.037993 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:49.037962 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:51.665003 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:51.664967 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:51.831614 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:51.831579 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:55.520594 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:55.520558 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:39:55.679039 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:39:55.679004 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:07.867006 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:07.866964 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:08.037572 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:08.037523 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:10.534102 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:10.534066 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:40:13.153840 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:40:13.153797 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:40:14.857935 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:14.857896 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l"] Nov 02 08:40:14.858361 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:14.858346 5899 cpu_manager.go:414] "containerMap: removing container" podUID="6d1ca139-ad15-4981-9265-b85548218656" containerName="revoke-keys" Nov 02 08:40:14.858361 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:14.858359 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1ca139-ad15-4981-9265-b85548218656" containerName="revoke-keys" Nov 02 08:40:14.858457 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:14.858447 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1ca139-ad15-4981-9265-b85548218656" containerName="revoke-keys" Nov 02 08:40:14.861552 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:14.861530 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" Nov 02 08:40:14.881084 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:14.881061 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l"] Nov 02 08:40:14.892358 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:14.892332 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:40:14.892495 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:14.892387 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:40:14.939457 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:14.939423 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-jlw7l\" (UID: \"7fe0319f-9584-4011-973b-fc72d4e171b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" Nov 02 08:40:14.939640 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:14.939465 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftqv\" (UniqueName: \"kubernetes.io/projected/7fe0319f-9584-4011-973b-fc72d4e171b8-kube-api-access-bftqv\") pod \"maas-api-revoke-sub-f904bff9-jlw7l\" (UID: \"7fe0319f-9584-4011-973b-fc72d4e171b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" Nov 02 08:40:14.939717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:14.939640 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-jlw7l\" (UID: \"7fe0319f-9584-4011-973b-fc72d4e171b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" Nov 02 08:40:15.040708 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:15.040668 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-jlw7l\" (UID: \"7fe0319f-9584-4011-973b-fc72d4e171b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" Nov 02 08:40:15.040887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:15.040755 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-jlw7l\" (UID: \"7fe0319f-9584-4011-973b-fc72d4e171b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" Nov 02 08:40:15.040887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:15.040791 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftqv\" (UniqueName: \"kubernetes.io/projected/7fe0319f-9584-4011-973b-fc72d4e171b8-kube-api-access-bftqv\") pod \"maas-api-revoke-sub-f904bff9-jlw7l\" (UID: \"7fe0319f-9584-4011-973b-fc72d4e171b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" Nov 02 08:40:15.041457 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:15.041424 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-jlw7l\" (UID: \"7fe0319f-9584-4011-973b-fc72d4e171b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" Nov 02 08:40:15.043422 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:15.043400 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-jlw7l\" (UID: \"7fe0319f-9584-4011-973b-fc72d4e171b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" Nov 02 08:40:15.065366 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:15.065332 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftqv\" (UniqueName: \"kubernetes.io/projected/7fe0319f-9584-4011-973b-fc72d4e171b8-kube-api-access-bftqv\") pod \"maas-api-revoke-sub-f904bff9-jlw7l\" (UID: \"7fe0319f-9584-4011-973b-fc72d4e171b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" Nov 02 08:40:15.174022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:15.173943 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" Nov 02 08:40:15.472534 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:15.472432 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l"] Nov 02 08:40:15.475287 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:40:15.475257 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe0319f_9584_4011_973b_fc72d4e171b8.slice/crio-ea3f6db0198f55d6403666593e3b91f3a7b0a23b94fe0f63d14f35656ebabe8e WatchSource:0}: Error finding container ea3f6db0198f55d6403666593e3b91f3a7b0a23b94fe0f63d14f35656ebabe8e: Status 404 returned error can't find the container with id ea3f6db0198f55d6403666593e3b91f3a7b0a23b94fe0f63d14f35656ebabe8e Nov 02 08:40:16.050005 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:16.049918 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" event={"ID":"7fe0319f-9584-4011-973b-fc72d4e171b8","Type":"ContainerStarted","Data":"851dadfdad965d455e9a1a3ad585d14003908875bd38b34efb43076e6e139e0d"} Nov 02 08:40:16.050005 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:16.049980 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" event={"ID":"7fe0319f-9584-4011-973b-fc72d4e171b8","Type":"ContainerStarted","Data":"ea3f6db0198f55d6403666593e3b91f3a7b0a23b94fe0f63d14f35656ebabe8e"} Nov 02 08:40:16.092896 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:16.092846 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" podStartSLOduration=2.092829684 podStartE2EDuration="2.092829684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:40:16.09166054 +0000 UTC m=+4516.389941366" watchObservedRunningTime="2026-11-02 08:40:16.092829684 +0000 UTC m=+4516.391110510" Nov 02 08:40:17.056207 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:17.056146 5899 generic.go:356] "Generic (PLEG): container finished" podID="7fe0319f-9584-4011-973b-fc72d4e171b8" containerID="851dadfdad965d455e9a1a3ad585d14003908875bd38b34efb43076e6e139e0d" exitCode=0 Nov 02 08:40:17.056593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:17.056234 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" event={"ID":"7fe0319f-9584-4011-973b-fc72d4e171b8","Type":"ContainerDied","Data":"851dadfdad965d455e9a1a3ad585d14003908875bd38b34efb43076e6e139e0d"} Nov 02 08:40:18.199595 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:18.199568 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" Nov 02 08:40:18.270001 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:18.269961 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe0319f-9584-4011-973b-fc72d4e171b8-kube-api-access-bftqv\" (UniqueName: \"kubernetes.io/projected/7fe0319f-9584-4011-973b-fc72d4e171b8-kube-api-access-bftqv\") pod \"7fe0319f-9584-4011-973b-fc72d4e171b8\" (UID: \"7fe0319f-9584-4011-973b-fc72d4e171b8\") " Nov 02 08:40:18.270197 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:18.270021 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-xks-ca\") pod \"7fe0319f-9584-4011-973b-fc72d4e171b8\" (UID: \"7fe0319f-9584-4011-973b-fc72d4e171b8\") " Nov 02 08:40:18.270291 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:18.270208 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-openshift-ca\") pod \"7fe0319f-9584-4011-973b-fc72d4e171b8\" (UID: \"7fe0319f-9584-4011-973b-fc72d4e171b8\") " Nov 02 08:40:18.270660 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:18.270627 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-openshift-ca" pod "7fe0319f-9584-4011-973b-fc72d4e171b8" (UID: "7fe0319f-9584-4011-973b-fc72d4e171b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:18.272059 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:18.272034 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-xks-ca" pod "7fe0319f-9584-4011-973b-fc72d4e171b8" (UID: "7fe0319f-9584-4011-973b-fc72d4e171b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:40:18.272213 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:18.272186 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe0319f-9584-4011-973b-fc72d4e171b8-kube-api-access-bftqv" pod "7fe0319f-9584-4011-973b-fc72d4e171b8" (UID: "7fe0319f-9584-4011-973b-fc72d4e171b8"). InnerVolumeSpecName "kube-api-access-bftqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:18.371741 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:18.371661 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:40:18.371741 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:18.371692 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe0319f-9584-4011-973b-fc72d4e171b8-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:40:18.371741 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:18.371703 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bftqv\" (UniqueName: \"kubernetes.io/projected/7fe0319f-9584-4011-973b-fc72d4e171b8-kube-api-access-bftqv\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:40:19.067773 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:19.067740 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" Nov 02 08:40:19.067773 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:19.067759 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l" event={"ID":"7fe0319f-9584-4011-973b-fc72d4e171b8","Type":"ContainerDied","Data":"ea3f6db0198f55d6403666593e3b91f3a7b0a23b94fe0f63d14f35656ebabe8e"} Nov 02 08:40:19.067988 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:19.067808 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea3f6db0198f55d6403666593e3b91f3a7b0a23b94fe0f63d14f35656ebabe8e" Nov 02 08:40:39.312393 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.312353 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65"] Nov 02 08:40:39.312801 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.312780 5899 cpu_manager.go:414] "containerMap: removing container" podUID="7fe0319f-9584-4011-973b-fc72d4e171b8" containerName="revoke-keys" Nov 02 08:40:39.312801 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.312791 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe0319f-9584-4011-973b-fc72d4e171b8" containerName="revoke-keys" Nov 02 08:40:39.312896 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.312886 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe0319f-9584-4011-973b-fc72d4e171b8" containerName="revoke-keys" Nov 02 08:40:39.316180 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.316144 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" Nov 02 08:40:39.333848 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.333786 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:40:39.336582 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.336553 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65"] Nov 02 08:40:39.346943 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.346908 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:40:39.348449 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.348420 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-4ct65\" (UID: \"a245ce8f-0097-49ce-869a-d74a1c024296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" Nov 02 08:40:39.348604 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.348477 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnsh7\" (UniqueName: \"kubernetes.io/projected/a245ce8f-0097-49ce-869a-d74a1c024296-kube-api-access-wnsh7\") pod \"maas-api-revoke-sub-f03629db-4ct65\" (UID: \"a245ce8f-0097-49ce-869a-d74a1c024296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" Nov 02 08:40:39.348604 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.348591 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-4ct65\" (UID: \"a245ce8f-0097-49ce-869a-d74a1c024296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" Nov 02 08:40:39.449392 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.449349 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-4ct65\" (UID: \"a245ce8f-0097-49ce-869a-d74a1c024296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" Nov 02 08:40:39.449573 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.449420 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-4ct65\" (UID: \"a245ce8f-0097-49ce-869a-d74a1c024296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" Nov 02 08:40:39.449573 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.449472 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnsh7\" (UniqueName: \"kubernetes.io/projected/a245ce8f-0097-49ce-869a-d74a1c024296-kube-api-access-wnsh7\") pod \"maas-api-revoke-sub-f03629db-4ct65\" (UID: \"a245ce8f-0097-49ce-869a-d74a1c024296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" Nov 02 08:40:39.450017 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.449993 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-4ct65\" (UID: \"a245ce8f-0097-49ce-869a-d74a1c024296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" Nov 02 08:40:39.451817 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.451789 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-4ct65\" (UID: \"a245ce8f-0097-49ce-869a-d74a1c024296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" Nov 02 08:40:39.473862 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.473829 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnsh7\" (UniqueName: \"kubernetes.io/projected/a245ce8f-0097-49ce-869a-d74a1c024296-kube-api-access-wnsh7\") pod \"maas-api-revoke-sub-f03629db-4ct65\" (UID: \"a245ce8f-0097-49ce-869a-d74a1c024296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" Nov 02 08:40:39.629667 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.629580 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" Nov 02 08:40:39.932410 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:39.932301 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65"] Nov 02 08:40:39.934836 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:40:39.934807 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda245ce8f_0097_49ce_869a_d74a1c024296.slice/crio-2a13280187fbb225d24f5a94a70c5e1ef67a1e87cf4912ad0d08e08031116dcf WatchSource:0}: Error finding container 2a13280187fbb225d24f5a94a70c5e1ef67a1e87cf4912ad0d08e08031116dcf: Status 404 returned error can't find the container with id 2a13280187fbb225d24f5a94a70c5e1ef67a1e87cf4912ad0d08e08031116dcf Nov 02 08:40:40.222069 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:40.221986 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" event={"ID":"a245ce8f-0097-49ce-869a-d74a1c024296","Type":"ContainerStarted","Data":"2a13280187fbb225d24f5a94a70c5e1ef67a1e87cf4912ad0d08e08031116dcf"} Nov 02 08:40:41.228077 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:41.228037 5899 generic.go:356] "Generic (PLEG): container finished" podID="a245ce8f-0097-49ce-869a-d74a1c024296" containerID="4ab4d9642582a50bdd45312f7fa0243eb4aa9dc99ea630e90ad320da537ddff7" exitCode=0 Nov 02 08:40:41.228553 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:41.228122 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" event={"ID":"a245ce8f-0097-49ce-869a-d74a1c024296","Type":"ContainerDied","Data":"4ab4d9642582a50bdd45312f7fa0243eb4aa9dc99ea630e90ad320da537ddff7"} Nov 02 08:40:42.366390 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:42.366362 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" Nov 02 08:40:42.374840 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:42.374810 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-openshift-ca\") pod \"a245ce8f-0097-49ce-869a-d74a1c024296\" (UID: \"a245ce8f-0097-49ce-869a-d74a1c024296\") " Nov 02 08:40:42.374946 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:42.374868 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a245ce8f-0097-49ce-869a-d74a1c024296-kube-api-access-wnsh7\" (UniqueName: \"kubernetes.io/projected/a245ce8f-0097-49ce-869a-d74a1c024296-kube-api-access-wnsh7\") pod \"a245ce8f-0097-49ce-869a-d74a1c024296\" (UID: \"a245ce8f-0097-49ce-869a-d74a1c024296\") " Nov 02 08:40:42.374946 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:42.374923 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-xks-ca\") pod \"a245ce8f-0097-49ce-869a-d74a1c024296\" (UID: \"a245ce8f-0097-49ce-869a-d74a1c024296\") " Nov 02 08:40:42.375210 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:42.375181 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-openshift-ca" pod "a245ce8f-0097-49ce-869a-d74a1c024296" (UID: "a245ce8f-0097-49ce-869a-d74a1c024296"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:42.376952 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:42.376927 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-xks-ca" pod "a245ce8f-0097-49ce-869a-d74a1c024296" (UID: "a245ce8f-0097-49ce-869a-d74a1c024296"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:40:42.377001 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:42.376933 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a245ce8f-0097-49ce-869a-d74a1c024296-kube-api-access-wnsh7" pod "a245ce8f-0097-49ce-869a-d74a1c024296" (UID: "a245ce8f-0097-49ce-869a-d74a1c024296"). InnerVolumeSpecName "kube-api-access-wnsh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:42.476678 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:42.476635 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:40:42.476678 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:42.476671 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnsh7\" (UniqueName: \"kubernetes.io/projected/a245ce8f-0097-49ce-869a-d74a1c024296-kube-api-access-wnsh7\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:40:42.476678 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:42.476682 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a245ce8f-0097-49ce-869a-d74a1c024296-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:40:43.139789 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:40:43.139755 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:40:43.240630 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:43.240592 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" event={"ID":"a245ce8f-0097-49ce-869a-d74a1c024296","Type":"ContainerDied","Data":"2a13280187fbb225d24f5a94a70c5e1ef67a1e87cf4912ad0d08e08031116dcf"} Nov 02 08:40:43.240827 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:43.240651 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a13280187fbb225d24f5a94a70c5e1ef67a1e87cf4912ad0d08e08031116dcf" Nov 02 08:40:43.240827 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:43.240717 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65" Nov 02 08:40:55.487094 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.487057 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx"] Nov 02 08:40:55.487538 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.487523 5899 cpu_manager.go:414] "containerMap: removing container" podUID="a245ce8f-0097-49ce-869a-d74a1c024296" containerName="revoke-keys" Nov 02 08:40:55.487607 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.487539 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a245ce8f-0097-49ce-869a-d74a1c024296" containerName="revoke-keys" Nov 02 08:40:55.487650 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.487608 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="a245ce8f-0097-49ce-869a-d74a1c024296" containerName="revoke-keys" Nov 02 08:40:55.491150 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.491131 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" Nov 02 08:40:55.510585 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.510505 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx"] Nov 02 08:40:55.521211 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.521134 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:40:55.521386 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.521273 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:40:55.542355 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.542321 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vgsbx\" (UID: \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" Nov 02 08:40:55.542551 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.542369 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vgsbx\" (UID: \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" Nov 02 08:40:55.542551 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.542514 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmhtr\" (UniqueName: \"kubernetes.io/projected/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-kube-api-access-rmhtr\") pod \"maas-api-revoke-sub-0165d7d2-vgsbx\" (UID: \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" Nov 02 08:40:55.643783 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.643742 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmhtr\" (UniqueName: \"kubernetes.io/projected/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-kube-api-access-rmhtr\") pod \"maas-api-revoke-sub-0165d7d2-vgsbx\" (UID: \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" Nov 02 08:40:55.643970 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.643798 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vgsbx\" (UID: \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" Nov 02 08:40:55.643970 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.643823 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vgsbx\" (UID: \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" Nov 02 08:40:55.644578 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.644547 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vgsbx\" (UID: \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" Nov 02 08:40:55.646078 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.646052 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vgsbx\" (UID: \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" Nov 02 08:40:55.668044 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.668008 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmhtr\" (UniqueName: \"kubernetes.io/projected/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-kube-api-access-rmhtr\") pod \"maas-api-revoke-sub-0165d7d2-vgsbx\" (UID: \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" Nov 02 08:40:55.808406 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:55.808316 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" Nov 02 08:40:56.109865 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:56.109840 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx"] Nov 02 08:40:56.112977 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:40:56.112948 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cabc0e3_6e2e_4b1f_a9b7_76db16312d68.slice/crio-744e6f0cb76ceb3b9b3236b9cd7757a6d427c8731af77660f80f290558195d90 WatchSource:0}: Error finding container 744e6f0cb76ceb3b9b3236b9cd7757a6d427c8731af77660f80f290558195d90: Status 404 returned error can't find the container with id 744e6f0cb76ceb3b9b3236b9cd7757a6d427c8731af77660f80f290558195d90 Nov 02 08:40:56.354904 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:56.354863 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" event={"ID":"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68","Type":"ContainerStarted","Data":"744e6f0cb76ceb3b9b3236b9cd7757a6d427c8731af77660f80f290558195d90"} Nov 02 08:40:57.361514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:57.361478 5899 generic.go:356] "Generic (PLEG): container finished" podID="7cabc0e3-6e2e-4b1f-a9b7-76db16312d68" containerID="7ccfd2b661dc4b102ae3842c5cc4c2ee6ac51e2419dcbff57f66beee364ef1fb" exitCode=0 Nov 02 08:40:57.361972 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:57.361541 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" event={"ID":"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68","Type":"ContainerDied","Data":"7ccfd2b661dc4b102ae3842c5cc4c2ee6ac51e2419dcbff57f66beee364ef1fb"} Nov 02 08:40:58.504446 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:58.504419 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" Nov 02 08:40:58.573558 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:58.573523 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-kube-api-access-rmhtr\" (UniqueName: \"kubernetes.io/projected/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-kube-api-access-rmhtr\") pod \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\" (UID: \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\") " Nov 02 08:40:58.573775 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:58.573647 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-openshift-ca\") pod \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\" (UID: \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\") " Nov 02 08:40:58.573775 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:58.573692 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-xks-ca\") pod \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\" (UID: \"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68\") " Nov 02 08:40:58.574122 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:58.574088 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-openshift-ca" pod "7cabc0e3-6e2e-4b1f-a9b7-76db16312d68" (UID: "7cabc0e3-6e2e-4b1f-a9b7-76db16312d68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:40:58.575876 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:58.575845 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-xks-ca" pod "7cabc0e3-6e2e-4b1f-a9b7-76db16312d68" (UID: "7cabc0e3-6e2e-4b1f-a9b7-76db16312d68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:40:58.575982 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:58.575904 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-kube-api-access-rmhtr" pod "7cabc0e3-6e2e-4b1f-a9b7-76db16312d68" (UID: "7cabc0e3-6e2e-4b1f-a9b7-76db16312d68"). InnerVolumeSpecName "kube-api-access-rmhtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:40:58.674932 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:58.674841 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:40:58.674932 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:58.674877 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:40:58.674932 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:58.674892 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmhtr\" (UniqueName: \"kubernetes.io/projected/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68-kube-api-access-rmhtr\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:40:59.371829 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:59.371797 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" Nov 02 08:40:59.371829 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:59.371810 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx" event={"ID":"7cabc0e3-6e2e-4b1f-a9b7-76db16312d68","Type":"ContainerDied","Data":"744e6f0cb76ceb3b9b3236b9cd7757a6d427c8731af77660f80f290558195d90"} Nov 02 08:40:59.372043 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:40:59.371856 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="744e6f0cb76ceb3b9b3236b9cd7757a6d427c8731af77660f80f290558195d90" Nov 02 08:41:06.568717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.568673 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8"] Nov 02 08:41:06.569329 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.569309 5899 cpu_manager.go:414] "containerMap: removing container" podUID="7cabc0e3-6e2e-4b1f-a9b7-76db16312d68" containerName="revoke-keys" Nov 02 08:41:06.569427 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.569331 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cabc0e3-6e2e-4b1f-a9b7-76db16312d68" containerName="revoke-keys" Nov 02 08:41:06.569509 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.569494 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cabc0e3-6e2e-4b1f-a9b7-76db16312d68" containerName="revoke-keys" Nov 02 08:41:06.572850 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.572827 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" Nov 02 08:41:06.592014 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.591980 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8"] Nov 02 08:41:06.603087 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.603049 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:41:06.603293 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.603109 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:41:06.757672 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.757622 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-5zcw8\" (UID: \"aab28fc7-3145-4597-87ec-8284ed8d507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" Nov 02 08:41:06.757672 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.757677 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lb4\" (UniqueName: \"kubernetes.io/projected/aab28fc7-3145-4597-87ec-8284ed8d507a-kube-api-access-64lb4\") pod \"maas-api-revoke-sub-6b19338f-5zcw8\" (UID: \"aab28fc7-3145-4597-87ec-8284ed8d507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" Nov 02 08:41:06.757931 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.757876 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-5zcw8\" (UID: \"aab28fc7-3145-4597-87ec-8284ed8d507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" Nov 02 08:41:06.858734 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.858636 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-5zcw8\" (UID: \"aab28fc7-3145-4597-87ec-8284ed8d507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" Nov 02 08:41:06.858734 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.858696 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-5zcw8\" (UID: \"aab28fc7-3145-4597-87ec-8284ed8d507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" Nov 02 08:41:06.858734 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.858719 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lb4\" (UniqueName: \"kubernetes.io/projected/aab28fc7-3145-4597-87ec-8284ed8d507a-kube-api-access-64lb4\") pod \"maas-api-revoke-sub-6b19338f-5zcw8\" (UID: \"aab28fc7-3145-4597-87ec-8284ed8d507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" Nov 02 08:41:06.859389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.859351 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-5zcw8\" (UID: \"aab28fc7-3145-4597-87ec-8284ed8d507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" Nov 02 08:41:06.861017 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.860986 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-5zcw8\" (UID: \"aab28fc7-3145-4597-87ec-8284ed8d507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" Nov 02 08:41:06.888139 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:06.888108 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lb4\" (UniqueName: \"kubernetes.io/projected/aab28fc7-3145-4597-87ec-8284ed8d507a-kube-api-access-64lb4\") pod \"maas-api-revoke-sub-6b19338f-5zcw8\" (UID: \"aab28fc7-3145-4597-87ec-8284ed8d507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" Nov 02 08:41:07.186341 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:07.186245 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" Nov 02 08:41:07.486351 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:07.486323 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8"] Nov 02 08:41:07.488765 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:41:07.488723 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab28fc7_3145_4597_87ec_8284ed8d507a.slice/crio-050245eb180977fd2ad92e0b5f308c504b5bbda2e2574a1142c83b724a5949a5 WatchSource:0}: Error finding container 050245eb180977fd2ad92e0b5f308c504b5bbda2e2574a1142c83b724a5949a5: Status 404 returned error can't find the container with id 050245eb180977fd2ad92e0b5f308c504b5bbda2e2574a1142c83b724a5949a5 Nov 02 08:41:08.417789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:08.417750 5899 generic.go:356] "Generic (PLEG): container finished" podID="aab28fc7-3145-4597-87ec-8284ed8d507a" containerID="b228f8ce43da83572ed37701c04dd488d82d5669580d9f938dab9418e08fe563" exitCode=0 Nov 02 08:41:08.418196 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:08.417789 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" event={"ID":"aab28fc7-3145-4597-87ec-8284ed8d507a","Type":"ContainerDied","Data":"b228f8ce43da83572ed37701c04dd488d82d5669580d9f938dab9418e08fe563"} Nov 02 08:41:08.418196 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:08.417851 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" event={"ID":"aab28fc7-3145-4597-87ec-8284ed8d507a","Type":"ContainerStarted","Data":"050245eb180977fd2ad92e0b5f308c504b5bbda2e2574a1142c83b724a5949a5"} Nov 02 08:41:09.554024 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:09.553989 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" Nov 02 08:41:09.687561 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:09.687479 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab28fc7-3145-4597-87ec-8284ed8d507a-kube-api-access-64lb4\" (UniqueName: \"kubernetes.io/projected/aab28fc7-3145-4597-87ec-8284ed8d507a-kube-api-access-64lb4\") pod \"aab28fc7-3145-4597-87ec-8284ed8d507a\" (UID: \"aab28fc7-3145-4597-87ec-8284ed8d507a\") " Nov 02 08:41:09.687561 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:09.687521 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-openshift-ca\") pod \"aab28fc7-3145-4597-87ec-8284ed8d507a\" (UID: \"aab28fc7-3145-4597-87ec-8284ed8d507a\") " Nov 02 08:41:09.687765 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:09.687621 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-xks-ca\") pod \"aab28fc7-3145-4597-87ec-8284ed8d507a\" (UID: \"aab28fc7-3145-4597-87ec-8284ed8d507a\") " Nov 02 08:41:09.688002 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:09.687976 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-openshift-ca" pod "aab28fc7-3145-4597-87ec-8284ed8d507a" (UID: "aab28fc7-3145-4597-87ec-8284ed8d507a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:41:09.689687 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:09.689667 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab28fc7-3145-4597-87ec-8284ed8d507a-kube-api-access-64lb4" pod "aab28fc7-3145-4597-87ec-8284ed8d507a" (UID: "aab28fc7-3145-4597-87ec-8284ed8d507a"). InnerVolumeSpecName "kube-api-access-64lb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:41:09.689795 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:09.689675 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-xks-ca" pod "aab28fc7-3145-4597-87ec-8284ed8d507a" (UID: "aab28fc7-3145-4597-87ec-8284ed8d507a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:41:09.788863 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:09.788810 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64lb4\" (UniqueName: \"kubernetes.io/projected/aab28fc7-3145-4597-87ec-8284ed8d507a-kube-api-access-64lb4\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:41:09.788863 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:09.788855 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:41:09.788863 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:09.788866 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab28fc7-3145-4597-87ec-8284ed8d507a-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:41:10.429048 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:10.429012 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" Nov 02 08:41:10.429048 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:10.429037 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8" event={"ID":"aab28fc7-3145-4597-87ec-8284ed8d507a","Type":"ContainerDied","Data":"050245eb180977fd2ad92e0b5f308c504b5bbda2e2574a1142c83b724a5949a5"} Nov 02 08:41:10.429282 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:10.429120 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="050245eb180977fd2ad92e0b5f308c504b5bbda2e2574a1142c83b724a5949a5" Nov 02 08:41:10.672449 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:10.672416 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:13.140213 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:41:13.140156 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:41:20.455674 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:20.455643 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:20.838863 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:20.838813 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:26.608181 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:41:26.608131 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:41:43.139213 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:41:43.139113 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:42:09.706353 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.706315 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj"] Nov 02 08:42:09.706862 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.706745 5899 cpu_manager.go:414] "containerMap: removing container" podUID="aab28fc7-3145-4597-87ec-8284ed8d507a" containerName="revoke-keys" Nov 02 08:42:09.706862 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.706757 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab28fc7-3145-4597-87ec-8284ed8d507a" containerName="revoke-keys" Nov 02 08:42:09.706862 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.706834 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab28fc7-3145-4597-87ec-8284ed8d507a" containerName="revoke-keys" Nov 02 08:42:09.710342 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.710321 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" Nov 02 08:42:09.732222 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.732187 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj"] Nov 02 08:42:09.742281 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.742252 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:42:09.742420 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.742314 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:42:09.780092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.780059 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxh5\" (UniqueName: \"kubernetes.io/projected/38228fca-dfc7-4041-ae75-5099d9946e8b-kube-api-access-znxh5\") pod \"maas-api-revoke-sub-ce062c03-lvtgj\" (UID: \"38228fca-dfc7-4041-ae75-5099d9946e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" Nov 02 08:42:09.780296 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.780098 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce062c03-lvtgj\" (UID: \"38228fca-dfc7-4041-ae75-5099d9946e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" Nov 02 08:42:09.780296 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.780122 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce062c03-lvtgj\" (UID: \"38228fca-dfc7-4041-ae75-5099d9946e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" Nov 02 08:42:09.880833 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.880790 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxh5\" (UniqueName: \"kubernetes.io/projected/38228fca-dfc7-4041-ae75-5099d9946e8b-kube-api-access-znxh5\") pod \"maas-api-revoke-sub-ce062c03-lvtgj\" (UID: \"38228fca-dfc7-4041-ae75-5099d9946e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" Nov 02 08:42:09.880833 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.880833 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce062c03-lvtgj\" (UID: \"38228fca-dfc7-4041-ae75-5099d9946e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" Nov 02 08:42:09.881130 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.880853 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce062c03-lvtgj\" (UID: \"38228fca-dfc7-4041-ae75-5099d9946e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" Nov 02 08:42:09.881554 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.881517 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce062c03-lvtgj\" (UID: \"38228fca-dfc7-4041-ae75-5099d9946e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" Nov 02 08:42:09.883195 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.883175 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce062c03-lvtgj\" (UID: \"38228fca-dfc7-4041-ae75-5099d9946e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" Nov 02 08:42:09.904896 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:09.904871 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxh5\" (UniqueName: \"kubernetes.io/projected/38228fca-dfc7-4041-ae75-5099d9946e8b-kube-api-access-znxh5\") pod \"maas-api-revoke-sub-ce062c03-lvtgj\" (UID: \"38228fca-dfc7-4041-ae75-5099d9946e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" Nov 02 08:42:10.023475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:10.023437 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" Nov 02 08:42:10.322809 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:10.322781 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj"] Nov 02 08:42:10.325246 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:42:10.325212 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38228fca_dfc7_4041_ae75_5099d9946e8b.slice/crio-f5163f19ce9c09ba59436773736ac90d063229d47ccb25532b3ded3def022b96 WatchSource:0}: Error finding container f5163f19ce9c09ba59436773736ac90d063229d47ccb25532b3ded3def022b96: Status 404 returned error can't find the container with id f5163f19ce9c09ba59436773736ac90d063229d47ccb25532b3ded3def022b96 Nov 02 08:42:10.776878 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:10.776837 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" event={"ID":"38228fca-dfc7-4041-ae75-5099d9946e8b","Type":"ContainerStarted","Data":"f6eaa7c52b395e758f685a1fe46833f145f0e20e6ed7771893b3c1f49b0700f2"} Nov 02 08:42:10.777384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:10.776896 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" event={"ID":"38228fca-dfc7-4041-ae75-5099d9946e8b","Type":"ContainerStarted","Data":"f5163f19ce9c09ba59436773736ac90d063229d47ccb25532b3ded3def022b96"} Nov 02 08:42:10.779753 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:10.779722 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:10.822475 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:10.822403 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" podStartSLOduration=1.822384585 podStartE2EDuration="1.822384585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:42:10.819019106 +0000 UTC m=+4631.117299933" watchObservedRunningTime="2026-11-02 08:42:10.822384585 +0000 UTC m=+4631.120665413" Nov 02 08:42:11.783841 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:11.783800 5899 generic.go:356] "Generic (PLEG): container finished" podID="38228fca-dfc7-4041-ae75-5099d9946e8b" containerID="f6eaa7c52b395e758f685a1fe46833f145f0e20e6ed7771893b3c1f49b0700f2" exitCode=0 Nov 02 08:42:11.784278 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:11.783892 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" event={"ID":"38228fca-dfc7-4041-ae75-5099d9946e8b","Type":"ContainerDied","Data":"f6eaa7c52b395e758f685a1fe46833f145f0e20e6ed7771893b3c1f49b0700f2"} Nov 02 08:42:12.923006 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:12.922978 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" Nov 02 08:42:13.010643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:13.010610 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-openshift-ca\") pod \"38228fca-dfc7-4041-ae75-5099d9946e8b\" (UID: \"38228fca-dfc7-4041-ae75-5099d9946e8b\") " Nov 02 08:42:13.010839 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:13.010711 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38228fca-dfc7-4041-ae75-5099d9946e8b-kube-api-access-znxh5\" (UniqueName: \"kubernetes.io/projected/38228fca-dfc7-4041-ae75-5099d9946e8b-kube-api-access-znxh5\") pod \"38228fca-dfc7-4041-ae75-5099d9946e8b\" (UID: \"38228fca-dfc7-4041-ae75-5099d9946e8b\") " Nov 02 08:42:13.010839 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:13.010767 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-xks-ca\") pod \"38228fca-dfc7-4041-ae75-5099d9946e8b\" (UID: \"38228fca-dfc7-4041-ae75-5099d9946e8b\") " Nov 02 08:42:13.011098 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:13.011072 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-openshift-ca" pod "38228fca-dfc7-4041-ae75-5099d9946e8b" (UID: "38228fca-dfc7-4041-ae75-5099d9946e8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:42:13.011209 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:13.011153 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:42:13.012872 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:13.012837 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-xks-ca" pod "38228fca-dfc7-4041-ae75-5099d9946e8b" (UID: "38228fca-dfc7-4041-ae75-5099d9946e8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:42:13.012973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:13.012880 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38228fca-dfc7-4041-ae75-5099d9946e8b-kube-api-access-znxh5" pod "38228fca-dfc7-4041-ae75-5099d9946e8b" (UID: "38228fca-dfc7-4041-ae75-5099d9946e8b"). InnerVolumeSpecName "kube-api-access-znxh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:13.112094 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:13.112010 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znxh5\" (UniqueName: \"kubernetes.io/projected/38228fca-dfc7-4041-ae75-5099d9946e8b-kube-api-access-znxh5\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:42:13.112094 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:13.112042 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38228fca-dfc7-4041-ae75-5099d9946e8b-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:42:13.140367 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:42:13.140325 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:42:13.795467 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:13.795435 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" Nov 02 08:42:13.795651 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:13.795471 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj" event={"ID":"38228fca-dfc7-4041-ae75-5099d9946e8b","Type":"ContainerDied","Data":"f5163f19ce9c09ba59436773736ac90d063229d47ccb25532b3ded3def022b96"} Nov 02 08:42:13.795651 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:13.795523 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5163f19ce9c09ba59436773736ac90d063229d47ccb25532b3ded3def022b96" Nov 02 08:42:21.678716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:21.678654 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:21.821512 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:21.821477 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:24.369613 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.369575 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz"] Nov 02 08:42:24.370139 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.370104 5899 cpu_manager.go:414] "containerMap: removing container" podUID="38228fca-dfc7-4041-ae75-5099d9946e8b" containerName="revoke-keys" Nov 02 08:42:24.370139 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.370123 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38228fca-dfc7-4041-ae75-5099d9946e8b" containerName="revoke-keys" Nov 02 08:42:24.370386 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.370215 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="38228fca-dfc7-4041-ae75-5099d9946e8b" containerName="revoke-keys" Nov 02 08:42:24.373593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.373573 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" Nov 02 08:42:24.390580 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.390552 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:42:24.403693 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.403657 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:42:24.405663 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.405636 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz"] Nov 02 08:42:24.429433 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.429398 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b72af5-m4tzz\" (UID: \"29171543-639c-4da8-ba3a-56656f8a9f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" Nov 02 08:42:24.429610 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.429455 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b72af5-m4tzz\" (UID: \"29171543-639c-4da8-ba3a-56656f8a9f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" Nov 02 08:42:24.429610 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.429568 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld567\" (UniqueName: \"kubernetes.io/projected/29171543-639c-4da8-ba3a-56656f8a9f9d-kube-api-access-ld567\") pod \"maas-api-revoke-sub-a8b72af5-m4tzz\" (UID: \"29171543-639c-4da8-ba3a-56656f8a9f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" Nov 02 08:42:24.530507 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.530466 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld567\" (UniqueName: \"kubernetes.io/projected/29171543-639c-4da8-ba3a-56656f8a9f9d-kube-api-access-ld567\") pod \"maas-api-revoke-sub-a8b72af5-m4tzz\" (UID: \"29171543-639c-4da8-ba3a-56656f8a9f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" Nov 02 08:42:24.530707 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.530585 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b72af5-m4tzz\" (UID: \"29171543-639c-4da8-ba3a-56656f8a9f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" Nov 02 08:42:24.530707 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.530632 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b72af5-m4tzz\" (UID: \"29171543-639c-4da8-ba3a-56656f8a9f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" Nov 02 08:42:24.531306 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.531283 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b72af5-m4tzz\" (UID: \"29171543-639c-4da8-ba3a-56656f8a9f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" Nov 02 08:42:24.533059 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.533032 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b72af5-m4tzz\" (UID: \"29171543-639c-4da8-ba3a-56656f8a9f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" Nov 02 08:42:24.554755 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.554712 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld567\" (UniqueName: \"kubernetes.io/projected/29171543-639c-4da8-ba3a-56656f8a9f9d-kube-api-access-ld567\") pod \"maas-api-revoke-sub-a8b72af5-m4tzz\" (UID: \"29171543-639c-4da8-ba3a-56656f8a9f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" Nov 02 08:42:24.687041 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.686954 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" Nov 02 08:42:24.987967 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:24.987942 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz"] Nov 02 08:42:24.990307 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:42:24.990282 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29171543_639c_4da8_ba3a_56656f8a9f9d.slice/crio-bffbad99a6c00cef05d2c8eea7dff1c7fe330037606189f0b9087a23c8043bbe WatchSource:0}: Error finding container bffbad99a6c00cef05d2c8eea7dff1c7fe330037606189f0b9087a23c8043bbe: Status 404 returned error can't find the container with id bffbad99a6c00cef05d2c8eea7dff1c7fe330037606189f0b9087a23c8043bbe Nov 02 08:42:25.863568 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:25.863526 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" event={"ID":"29171543-639c-4da8-ba3a-56656f8a9f9d","Type":"ContainerStarted","Data":"ddd70a24ed6b8fa89e822255d3b7a2eaa105314c23e34ebd66592dd228b6e5ec"} Nov 02 08:42:25.863997 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:25.863592 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" event={"ID":"29171543-639c-4da8-ba3a-56656f8a9f9d","Type":"ContainerStarted","Data":"bffbad99a6c00cef05d2c8eea7dff1c7fe330037606189f0b9087a23c8043bbe"} Nov 02 08:42:25.910484 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:25.907471 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" podStartSLOduration=1.9074524130000001 podStartE2EDuration="1.907452413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:42:25.905759826 +0000 UTC m=+4646.204040651" watchObservedRunningTime="2026-11-02 08:42:25.907452413 +0000 UTC m=+4646.205733239" Nov 02 08:42:26.891546 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:26.891056 5899 generic.go:356] "Generic (PLEG): container finished" podID="29171543-639c-4da8-ba3a-56656f8a9f9d" containerID="ddd70a24ed6b8fa89e822255d3b7a2eaa105314c23e34ebd66592dd228b6e5ec" exitCode=0 Nov 02 08:42:26.891546 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:26.891148 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" event={"ID":"29171543-639c-4da8-ba3a-56656f8a9f9d","Type":"ContainerDied","Data":"ddd70a24ed6b8fa89e822255d3b7a2eaa105314c23e34ebd66592dd228b6e5ec"} Nov 02 08:42:28.028191 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:28.028144 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" Nov 02 08:42:28.199504 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:28.199419 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29171543-639c-4da8-ba3a-56656f8a9f9d-kube-api-access-ld567\" (UniqueName: \"kubernetes.io/projected/29171543-639c-4da8-ba3a-56656f8a9f9d-kube-api-access-ld567\") pod \"29171543-639c-4da8-ba3a-56656f8a9f9d\" (UID: \"29171543-639c-4da8-ba3a-56656f8a9f9d\") " Nov 02 08:42:28.199664 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:28.199540 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-xks-ca\") pod \"29171543-639c-4da8-ba3a-56656f8a9f9d\" (UID: \"29171543-639c-4da8-ba3a-56656f8a9f9d\") " Nov 02 08:42:28.199664 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:28.199602 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-openshift-ca\") pod \"29171543-639c-4da8-ba3a-56656f8a9f9d\" (UID: \"29171543-639c-4da8-ba3a-56656f8a9f9d\") " Nov 02 08:42:28.199961 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:28.199935 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-openshift-ca" pod "29171543-639c-4da8-ba3a-56656f8a9f9d" (UID: "29171543-639c-4da8-ba3a-56656f8a9f9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:42:28.201534 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:28.201509 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-xks-ca" pod "29171543-639c-4da8-ba3a-56656f8a9f9d" (UID: "29171543-639c-4da8-ba3a-56656f8a9f9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:42:28.201680 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:28.201660 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29171543-639c-4da8-ba3a-56656f8a9f9d-kube-api-access-ld567" pod "29171543-639c-4da8-ba3a-56656f8a9f9d" (UID: "29171543-639c-4da8-ba3a-56656f8a9f9d"). InnerVolumeSpecName "kube-api-access-ld567". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:42:28.300539 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:28.300503 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:42:28.300539 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:28.300535 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29171543-639c-4da8-ba3a-56656f8a9f9d-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:42:28.300539 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:28.300547 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld567\" (UniqueName: \"kubernetes.io/projected/29171543-639c-4da8-ba3a-56656f8a9f9d-kube-api-access-ld567\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:42:28.903421 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:28.903379 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" event={"ID":"29171543-639c-4da8-ba3a-56656f8a9f9d","Type":"ContainerDied","Data":"bffbad99a6c00cef05d2c8eea7dff1c7fe330037606189f0b9087a23c8043bbe"} Nov 02 08:42:28.903609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:28.903437 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bffbad99a6c00cef05d2c8eea7dff1c7fe330037606189f0b9087a23c8043bbe" Nov 02 08:42:28.903609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:28.903400 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz" Nov 02 08:42:36.373381 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:36.373345 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:36.881501 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:36.881464 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:37.018510 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:37.018475 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:40.092111 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:40.092077 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:40.233651 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:40.233599 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:41.666929 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.666888 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw"] Nov 02 08:42:41.667447 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.667360 5899 cpu_manager.go:414] "containerMap: removing container" podUID="29171543-639c-4da8-ba3a-56656f8a9f9d" containerName="revoke-keys" Nov 02 08:42:41.667447 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.667374 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29171543-639c-4da8-ba3a-56656f8a9f9d" containerName="revoke-keys" Nov 02 08:42:41.667523 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.667483 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="29171543-639c-4da8-ba3a-56656f8a9f9d" containerName="revoke-keys" Nov 02 08:42:41.670678 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.670658 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:41.710244 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.710209 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw"] Nov 02 08:42:41.710593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.710563 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-1b623b-serving-cert\"" Nov 02 08:42:41.710749 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.710623 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 08:42:41.710749 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.710735 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4tgq8\"" Nov 02 08:42:41.782204 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.782148 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:41.782384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.782211 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:41.782384 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.782357 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rdwb\" (UniqueName: \"kubernetes.io/projected/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-kube-api-access-2rdwb\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:41.782502 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.782396 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:41.883506 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.883463 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:41.883721 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.883543 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:41.883721 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.883569 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:41.883721 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.883649 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rdwb\" (UniqueName: \"kubernetes.io/projected/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-kube-api-access-2rdwb\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:41.884345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.884313 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:41.886156 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.886125 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:41.886318 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.886192 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:41.886439 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.886414 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:41.910554 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.910526 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rdwb\" (UniqueName: \"kubernetes.io/projected/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-kube-api-access-2rdwb\") pod \"maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:41.983526 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:41.983496 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:42.038053 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:42.037979 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:42.284762 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:42.284686 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw"] Nov 02 08:42:42.287361 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:42:42.287330 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fafbd9_73ac_414c_9da2_c31b071a9e0d.slice/crio-eb7a43cc2fb6eac6ae62c6b5e8d360081156d58aa1ae0c339decc43e20f69804 WatchSource:0}: Error finding container eb7a43cc2fb6eac6ae62c6b5e8d360081156d58aa1ae0c339decc43e20f69804: Status 404 returned error can't find the container with id eb7a43cc2fb6eac6ae62c6b5e8d360081156d58aa1ae0c339decc43e20f69804 Nov 02 08:42:42.288572 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:42.288555 5899 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:42:42.981930 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:42.981889 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" event={"ID":"a7fafbd9-73ac-414c-9da2-c31b071a9e0d","Type":"ContainerStarted","Data":"f0225760f296837088396922a7602c6d0f3e4e934ba5f47c66025287f478db4d"} Nov 02 08:42:42.982444 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:42.981953 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:42.982444 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:42.981980 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" event={"ID":"a7fafbd9-73ac-414c-9da2-c31b071a9e0d","Type":"ContainerStarted","Data":"eb7a43cc2fb6eac6ae62c6b5e8d360081156d58aa1ae0c339decc43e20f69804"} Nov 02 08:42:43.036523 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:43.036461 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" podStartSLOduration=1.662080209 podStartE2EDuration="2.036442169s" totalImagesPullingTime="374.36196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:42:43.034485448 +0000 UTC m=+4663.332766274" watchObservedRunningTime="2026-11-02 08:42:43.036442169 +0000 UTC m=+4663.334722995" Nov 02 08:42:43.141134 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:42:43.141098 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:42:43.827319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:43.827281 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:45.717294 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:45.717261 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:47.522446 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:47.522386 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:48.992237 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:48.992206 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:42:49.294765 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:49.294675 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:51.045447 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:51.045407 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:52.819486 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:52.819437 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:54.550305 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:54.550245 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:54.883667 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:54.883587 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:56.340858 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:56.340794 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:57.039640 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:57.039606 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb"] Nov 02 08:42:57.053009 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:57.052978 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df106b94-2rmvb"] Nov 02 08:42:57.818148 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:57.818115 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9420fdfa-e0b9-4a66-bbf6-1dbf986ad567" path="/var/lib/kubelet/pods/9420fdfa-e0b9-4a66-bbf6-1dbf986ad567/volumes" Nov 02 08:42:58.016990 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:58.016956 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:42:59.822880 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:42:59.822844 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:01.513340 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:01.513304 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:03.366846 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:03.366807 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:03.545784 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:03.545745 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:05.008120 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:05.008089 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:05.533482 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:05.533443 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:05.908609 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:05.908521 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:06.695618 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.695584 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk"] Nov 02 08:43:06.700486 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.700464 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.718790 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.718766 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1b623b\"/\"default-dockercfg-rrf8h\"" Nov 02 08:43:06.731971 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.731945 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1b623b\"/\"simulator-single-main-1b623b-kserve-self-signed-certs\"" Nov 02 08:43:06.755770 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.755738 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk"] Nov 02 08:43:06.770319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.770290 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1b623b\"/\"openshift-service-ca.crt\"" Nov 02 08:43:06.770319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.770311 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1b623b\"/\"kube-root-ca.crt\"" Nov 02 08:43:06.830447 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.830413 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:06.837581 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.837548 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-tmp-dir\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.837736 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.837596 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-home\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.837736 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.837661 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9576514-b918-4287-b0e9-0d1c46414ee0-tls-certs\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.837736 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.837686 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwj78\" (UniqueName: \"kubernetes.io/projected/f9576514-b918-4287-b0e9-0d1c46414ee0-kube-api-access-bwj78\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.837876 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.837775 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-dshm\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.837876 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.837871 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-model-cache\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.938752 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.938706 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-model-cache\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.938985 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.938780 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-tmp-dir\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.938985 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.938856 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-home\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.939237 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.939209 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-model-cache\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.939322 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.939242 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-tmp-dir\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.939396 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.939316 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-home\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.939461 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.939410 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9576514-b918-4287-b0e9-0d1c46414ee0-tls-certs\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.939525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.939467 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwj78\" (UniqueName: \"kubernetes.io/projected/f9576514-b918-4287-b0e9-0d1c46414ee0-kube-api-access-bwj78\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.939706 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.939683 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-dshm\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.941967 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.941940 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-dshm\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.942122 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.942102 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9576514-b918-4287-b0e9-0d1c46414ee0-tls-certs\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:06.965017 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:06.964950 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwj78\" (UniqueName: \"kubernetes.io/projected/f9576514-b918-4287-b0e9-0d1c46414ee0-kube-api-access-bwj78\") pod \"simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:07.012620 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:07.012583 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:07.223188 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:07.223075 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:07.315764 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:07.315735 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk"] Nov 02 08:43:07.318558 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:43:07.318533 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9576514_b918_4287_b0e9_0d1c46414ee0.slice/crio-5e161a3fa42af686d93b32e9b1d34de52784b384c0a2171f3e8f30de0838f8c3 WatchSource:0}: Error finding container 5e161a3fa42af686d93b32e9b1d34de52784b384c0a2171f3e8f30de0838f8c3: Status 404 returned error can't find the container with id 5e161a3fa42af686d93b32e9b1d34de52784b384c0a2171f3e8f30de0838f8c3 Nov 02 08:43:08.124704 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:08.124665 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" event={"ID":"f9576514-b918-4287-b0e9-0d1c46414ee0","Type":"ContainerStarted","Data":"e4869bd68aa514b5abd60e653ed5e5f148d5cb3e5c6e2f8a7ea4c40571da90d4"} Nov 02 08:43:08.125112 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:08.124715 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" event={"ID":"f9576514-b918-4287-b0e9-0d1c46414ee0","Type":"ContainerStarted","Data":"5e161a3fa42af686d93b32e9b1d34de52784b384c0a2171f3e8f30de0838f8c3"} Nov 02 08:43:09.012853 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:09.012817 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:09.117529 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:09.117492 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:10.899741 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:10.899658 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:11.043425 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:11.043390 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:12.623816 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:12.623775 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:13.141397 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:43:13.141361 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:43:14.327623 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:14.327589 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:16.184514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:16.184480 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:17.013184 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:17.013126 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:17.028415 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:17.028385 5899 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:17.081486 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:17.081413 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" podStartSLOduration=11.081396062 podStartE2EDuration="11.081396062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:43:08.17303289 +0000 UTC m=+4688.471313717" watchObservedRunningTime="2026-11-02 08:43:17.081396062 +0000 UTC m=+4697.379676888" Nov 02 08:43:17.185354 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:17.185324 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:43:17.924852 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:17.924810 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:18.063878 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:18.063843 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:19.688316 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:19.688255 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:19.953490 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:19.953365 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:21.397857 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:21.397819 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:23.207844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:23.207790 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:25.019419 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:25.019383 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:26.823103 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:26.823072 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:28.599788 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:28.599750 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:30.311253 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:30.311208 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:32.104305 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:32.104270 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:33.901306 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:33.901270 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:35.640895 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:35.640855 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:37.413844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:37.413804 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:39.095968 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:39.095912 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:40.812457 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:40.812423 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:42.699295 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:42.699258 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:43.140517 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:43:43.140482 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:43:44.509417 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:44.509379 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:46.306518 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:46.306480 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:48.005944 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:48.005908 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:49.694270 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:49.694236 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:50.424546 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:50.424504 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:50.902184 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:50.902127 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:51.406902 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:51.406867 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:53.186734 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:53.186689 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:54.901500 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:54.901463 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:55.352098 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:55.352063 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk"] Nov 02 08:43:55.448530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:55.448499 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:56.618381 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:56.618346 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:56.976211 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:56.976094 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:57.128934 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:57.128896 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:57.436676 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:57.436636 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:58.390595 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:58.390559 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:43:59.741928 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:59.741887 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw"] Nov 02 08:43:59.742380 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:59.742195 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" podUID="a7fafbd9-73ac-414c-9da2-c31b071a9e0d" containerName="maas-api" containerID="cri-o://f0225760f296837088396922a7602c6d0f3e4e934ba5f47c66025287f478db4d" gracePeriod=30 Nov 02 08:43:59.837348 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:43:59.837314 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:00.200913 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.200880 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:44:00.262271 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.262237 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-maas-api-tls\") pod \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " Nov 02 08:44:00.262428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.262362 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-openshift-service-ca\") pod \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " Nov 02 08:44:00.262428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.262415 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-kube-api-access-2rdwb\" (UniqueName: \"kubernetes.io/projected/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-kube-api-access-2rdwb\") pod \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " Nov 02 08:44:00.262528 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.262478 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-metrics-certs\") pod \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\" (UID: \"a7fafbd9-73ac-414c-9da2-c31b071a9e0d\") " Nov 02 08:44:00.262754 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.262715 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-openshift-service-ca" pod "a7fafbd9-73ac-414c-9da2-c31b071a9e0d" (UID: "a7fafbd9-73ac-414c-9da2-c31b071a9e0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:44:00.262898 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.262819 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:44:00.264417 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.264385 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-maas-api-tls" pod "a7fafbd9-73ac-414c-9da2-c31b071a9e0d" (UID: "a7fafbd9-73ac-414c-9da2-c31b071a9e0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:44:00.264564 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.264544 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-metrics-certs" pod "a7fafbd9-73ac-414c-9da2-c31b071a9e0d" (UID: "a7fafbd9-73ac-414c-9da2-c31b071a9e0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:44:00.264923 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.264894 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-kube-api-access-2rdwb" pod "a7fafbd9-73ac-414c-9da2-c31b071a9e0d" (UID: "a7fafbd9-73ac-414c-9da2-c31b071a9e0d"). InnerVolumeSpecName "kube-api-access-2rdwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:00.364320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.364236 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:44:00.364320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.364267 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rdwb\" (UniqueName: \"kubernetes.io/projected/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-kube-api-access-2rdwb\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:44:00.364320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.364277 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7fafbd9-73ac-414c-9da2-c31b071a9e0d-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:44:00.402735 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.402698 5899 generic.go:356] "Generic (PLEG): container finished" podID="a7fafbd9-73ac-414c-9da2-c31b071a9e0d" containerID="f0225760f296837088396922a7602c6d0f3e4e934ba5f47c66025287f478db4d" exitCode=0 Nov 02 08:44:00.402877 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.402750 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" event={"ID":"a7fafbd9-73ac-414c-9da2-c31b071a9e0d","Type":"ContainerDied","Data":"f0225760f296837088396922a7602c6d0f3e4e934ba5f47c66025287f478db4d"} Nov 02 08:44:00.402877 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.402790 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" event={"ID":"a7fafbd9-73ac-414c-9da2-c31b071a9e0d","Type":"ContainerDied","Data":"eb7a43cc2fb6eac6ae62c6b5e8d360081156d58aa1ae0c339decc43e20f69804"} Nov 02 08:44:00.402877 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.402812 5899 scope.go:122] "RemoveContainer" containerID="f0225760f296837088396922a7602c6d0f3e4e934ba5f47c66025287f478db4d" Nov 02 08:44:00.402877 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.402748 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw" Nov 02 08:44:00.412872 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.412851 5899 scope.go:122] "RemoveContainer" containerID="f0225760f296837088396922a7602c6d0f3e4e934ba5f47c66025287f478db4d" Nov 02 08:44:00.413242 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:44:00.413215 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0225760f296837088396922a7602c6d0f3e4e934ba5f47c66025287f478db4d\": container with ID starting with f0225760f296837088396922a7602c6d0f3e4e934ba5f47c66025287f478db4d not found: ID does not exist" containerID="f0225760f296837088396922a7602c6d0f3e4e934ba5f47c66025287f478db4d" Nov 02 08:44:00.413320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.413256 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0225760f296837088396922a7602c6d0f3e4e934ba5f47c66025287f478db4d"} err="failed to get container status \"f0225760f296837088396922a7602c6d0f3e4e934ba5f47c66025287f478db4d\": rpc error: code = NotFound desc = could not find container \"f0225760f296837088396922a7602c6d0f3e4e934ba5f47c66025287f478db4d\": container with ID starting with f0225760f296837088396922a7602c6d0f3e4e934ba5f47c66025287f478db4d not found: ID does not exist" Nov 02 08:44:00.457940 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.457909 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw"] Nov 02 08:44:00.461716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:00.461690 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1b623b-7c847bcddf-dj5fw"] Nov 02 08:44:01.320764 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:01.320727 5899 scope.go:122] "RemoveContainer" containerID="f82922b6075734289bd61c04a914e97e141d421c95970a04d4ae8164aefb1d84" Nov 02 08:44:01.818985 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:01.818953 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7fafbd9-73ac-414c-9da2-c31b071a9e0d" path="/var/lib/kubelet/pods/a7fafbd9-73ac-414c-9da2-c31b071a9e0d/volumes" Nov 02 08:44:03.183372 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:03.183341 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:03.338466 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:03.338436 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:10.372196 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.372122 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" podUID="f9576514-b918-4287-b0e9-0d1c46414ee0" containerName="main" containerID="cri-o://e4869bd68aa514b5abd60e653ed5e5f148d5cb3e5c6e2f8a7ea4c40571da90d4" gracePeriod=45 Nov 02 08:44:10.617516 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.617476 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:44:10.769264 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.769229 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-home\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-home\") pod \"f9576514-b918-4287-b0e9-0d1c46414ee0\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " Nov 02 08:44:10.769442 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.769281 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-tmp-dir\") pod \"f9576514-b918-4287-b0e9-0d1c46414ee0\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " Nov 02 08:44:10.769442 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.769340 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-model-cache\") pod \"f9576514-b918-4287-b0e9-0d1c46414ee0\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " Nov 02 08:44:10.769442 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.769360 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-dshm\") pod \"f9576514-b918-4287-b0e9-0d1c46414ee0\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " Nov 02 08:44:10.769633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.769475 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9576514-b918-4287-b0e9-0d1c46414ee0-kube-api-access-bwj78\" (UniqueName: \"kubernetes.io/projected/f9576514-b918-4287-b0e9-0d1c46414ee0-kube-api-access-bwj78\") pod \"f9576514-b918-4287-b0e9-0d1c46414ee0\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " Nov 02 08:44:10.769633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.769494 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-home" pod "f9576514-b918-4287-b0e9-0d1c46414ee0" (UID: "f9576514-b918-4287-b0e9-0d1c46414ee0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:10.769633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.769550 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9576514-b918-4287-b0e9-0d1c46414ee0-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9576514-b918-4287-b0e9-0d1c46414ee0-tls-certs\") pod \"f9576514-b918-4287-b0e9-0d1c46414ee0\" (UID: \"f9576514-b918-4287-b0e9-0d1c46414ee0\") " Nov 02 08:44:10.769633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.769568 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-tmp-dir" pod "f9576514-b918-4287-b0e9-0d1c46414ee0" (UID: "f9576514-b918-4287-b0e9-0d1c46414ee0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:10.769633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.769586 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-model-cache" pod "f9576514-b918-4287-b0e9-0d1c46414ee0" (UID: "f9576514-b918-4287-b0e9-0d1c46414ee0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:10.769900 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.769812 5899 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-home\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:44:10.769900 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.769833 5899 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-tmp-dir\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:44:10.769900 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.769846 5899 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-model-cache\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:44:10.771903 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.771873 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9576514-b918-4287-b0e9-0d1c46414ee0-tls-certs" pod "f9576514-b918-4287-b0e9-0d1c46414ee0" (UID: "f9576514-b918-4287-b0e9-0d1c46414ee0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:44:10.772061 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.771916 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-dshm" pod "f9576514-b918-4287-b0e9-0d1c46414ee0" (UID: "f9576514-b918-4287-b0e9-0d1c46414ee0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 08:44:10.772287 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.772267 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9576514-b918-4287-b0e9-0d1c46414ee0-kube-api-access-bwj78" pod "f9576514-b918-4287-b0e9-0d1c46414ee0" (UID: "f9576514-b918-4287-b0e9-0d1c46414ee0"). InnerVolumeSpecName "kube-api-access-bwj78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:10.870842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.870802 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwj78\" (UniqueName: \"kubernetes.io/projected/f9576514-b918-4287-b0e9-0d1c46414ee0-kube-api-access-bwj78\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:44:10.870842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.870834 5899 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9576514-b918-4287-b0e9-0d1c46414ee0-tls-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:44:10.870842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:10.870846 5899 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9576514-b918-4287-b0e9-0d1c46414ee0-dshm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:44:11.017553 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:11.017518 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:11.462232 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:11.462190 5899 generic.go:356] "Generic (PLEG): container finished" podID="f9576514-b918-4287-b0e9-0d1c46414ee0" containerID="e4869bd68aa514b5abd60e653ed5e5f148d5cb3e5c6e2f8a7ea4c40571da90d4" exitCode=0 Nov 02 08:44:11.462716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:11.462234 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" Nov 02 08:44:11.462716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:11.462263 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" event={"ID":"f9576514-b918-4287-b0e9-0d1c46414ee0","Type":"ContainerDied","Data":"e4869bd68aa514b5abd60e653ed5e5f148d5cb3e5c6e2f8a7ea4c40571da90d4"} Nov 02 08:44:11.462716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:11.462308 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk" event={"ID":"f9576514-b918-4287-b0e9-0d1c46414ee0","Type":"ContainerDied","Data":"5e161a3fa42af686d93b32e9b1d34de52784b384c0a2171f3e8f30de0838f8c3"} Nov 02 08:44:11.462716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:11.462325 5899 scope.go:122] "RemoveContainer" containerID="e4869bd68aa514b5abd60e653ed5e5f148d5cb3e5c6e2f8a7ea4c40571da90d4" Nov 02 08:44:11.472699 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:11.472676 5899 scope.go:122] "RemoveContainer" containerID="e4869bd68aa514b5abd60e653ed5e5f148d5cb3e5c6e2f8a7ea4c40571da90d4" Nov 02 08:44:11.472951 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:44:11.472929 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4869bd68aa514b5abd60e653ed5e5f148d5cb3e5c6e2f8a7ea4c40571da90d4\": container with ID starting with e4869bd68aa514b5abd60e653ed5e5f148d5cb3e5c6e2f8a7ea4c40571da90d4 not found: ID does not exist" containerID="e4869bd68aa514b5abd60e653ed5e5f148d5cb3e5c6e2f8a7ea4c40571da90d4" Nov 02 08:44:11.473033 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:11.472960 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4869bd68aa514b5abd60e653ed5e5f148d5cb3e5c6e2f8a7ea4c40571da90d4"} err="failed to get container status \"e4869bd68aa514b5abd60e653ed5e5f148d5cb3e5c6e2f8a7ea4c40571da90d4\": rpc error: code = NotFound desc = could not find container \"e4869bd68aa514b5abd60e653ed5e5f148d5cb3e5c6e2f8a7ea4c40571da90d4\": container with ID starting with e4869bd68aa514b5abd60e653ed5e5f148d5cb3e5c6e2f8a7ea4c40571da90d4 not found: ID does not exist" Nov 02 08:44:11.517191 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:11.517141 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk"] Nov 02 08:44:11.530124 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:11.530094 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-1b623b/simulator-single-main-1b623b-kserve-5bddd7b6d4-fnwdk"] Nov 02 08:44:11.818840 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:11.818802 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9576514-b918-4287-b0e9-0d1c46414ee0" path="/var/lib/kubelet/pods/f9576514-b918-4287-b0e9-0d1c46414ee0/volumes" Nov 02 08:44:12.441081 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:12.441045 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:13.141530 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:44:13.141487 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:44:30.481711 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:30.481675 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:30.637082 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:30.637046 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:31.995179 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:31.995129 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:32.158998 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:32.158957 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:32.293863 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:32.293782 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:33.489886 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:33.489833 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:34.117045 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:34.116988 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:34.284823 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:34.284787 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:36.232372 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:36.232298 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:38.000504 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:38.000457 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:39.029806 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:39.029761 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm"] Nov 02 08:44:39.042693 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:39.042656 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-glbtm"] Nov 02 08:44:39.798105 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:39.798045 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:39.834495 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:39.834461 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d1ca139-ad15-4981-9265-b85548218656" path="/var/lib/kubelet/pods/6d1ca139-ad15-4981-9265-b85548218656/volumes" Nov 02 08:44:40.501298 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:40.501246 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:40.674297 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:40.674223 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:41.365070 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.365033 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5"] Nov 02 08:44:41.365602 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.365581 5899 cpu_manager.go:414] "containerMap: removing container" podUID="a7fafbd9-73ac-414c-9da2-c31b071a9e0d" containerName="maas-api" Nov 02 08:44:41.365701 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.365603 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7fafbd9-73ac-414c-9da2-c31b071a9e0d" containerName="maas-api" Nov 02 08:44:41.365701 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.365651 5899 cpu_manager.go:414] "containerMap: removing container" podUID="f9576514-b918-4287-b0e9-0d1c46414ee0" containerName="main" Nov 02 08:44:41.365701 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.365660 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9576514-b918-4287-b0e9-0d1c46414ee0" containerName="main" Nov 02 08:44:41.365861 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.365783 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7fafbd9-73ac-414c-9da2-c31b071a9e0d" containerName="maas-api" Nov 02 08:44:41.365861 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.365801 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9576514-b918-4287-b0e9-0d1c46414ee0" containerName="main" Nov 02 08:44:41.369141 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.369119 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" Nov 02 08:44:41.394129 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.394102 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:44:41.403471 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.403443 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:44:41.405345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.405319 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5"] Nov 02 08:44:41.474119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.474085 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5\" (UID: \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" Nov 02 08:44:41.474331 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.474131 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5\" (UID: \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" Nov 02 08:44:41.474331 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.474183 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2lnd\" (UniqueName: \"kubernetes.io/projected/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-kube-api-access-x2lnd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5\" (UID: \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" Nov 02 08:44:41.575208 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.575128 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5\" (UID: \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" Nov 02 08:44:41.575615 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.575233 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5\" (UID: \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" Nov 02 08:44:41.575615 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.575266 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2lnd\" (UniqueName: \"kubernetes.io/projected/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-kube-api-access-x2lnd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5\" (UID: \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" Nov 02 08:44:41.575968 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.575946 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5\" (UID: \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" Nov 02 08:44:41.577451 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.577422 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5\" (UID: \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" Nov 02 08:44:41.600257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.600222 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2lnd\" (UniqueName: \"kubernetes.io/projected/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-kube-api-access-x2lnd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5\" (UID: \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" Nov 02 08:44:41.650956 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.650804 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:41.682254 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.682223 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" Nov 02 08:44:41.994988 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:41.994961 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5"] Nov 02 08:44:41.997757 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:44:41.997730 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9ca014_5a94_44a7_9b0e_e2ad9f24a0bd.slice/crio-1a86f294d0e3e237205ee17c28970d3a5ef462258f866b5d99eef15cf3f092a7 WatchSource:0}: Error finding container 1a86f294d0e3e237205ee17c28970d3a5ef462258f866b5d99eef15cf3f092a7: Status 404 returned error can't find the container with id 1a86f294d0e3e237205ee17c28970d3a5ef462258f866b5d99eef15cf3f092a7 Nov 02 08:44:42.621439 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:42.621348 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" event={"ID":"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd","Type":"ContainerStarted","Data":"dddfccfaeba6e25d48c3c8e4f8de421c52740323a9932df2df57d72936db283f"} Nov 02 08:44:42.621439 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:42.621404 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" event={"ID":"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd","Type":"ContainerStarted","Data":"1a86f294d0e3e237205ee17c28970d3a5ef462258f866b5d99eef15cf3f092a7"} Nov 02 08:44:42.663936 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:42.663885 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" podStartSLOduration=1.663870975 podStartE2EDuration="1.663870975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:44:42.663224301 +0000 UTC m=+4782.961505126" watchObservedRunningTime="2026-11-02 08:44:42.663870975 +0000 UTC m=+4782.962151800" Nov 02 08:44:43.140245 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:44:43.140208 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:44:43.454824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:43.454792 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:43.627931 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:43.627893 5899 generic.go:356] "Generic (PLEG): container finished" podID="ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd" containerID="dddfccfaeba6e25d48c3c8e4f8de421c52740323a9932df2df57d72936db283f" exitCode=0 Nov 02 08:44:43.627931 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:43.627937 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" event={"ID":"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd","Type":"ContainerDied","Data":"dddfccfaeba6e25d48c3c8e4f8de421c52740323a9932df2df57d72936db283f"} Nov 02 08:44:44.769268 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:44.769241 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" Nov 02 08:44:44.905724 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:44.905637 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-kube-api-access-x2lnd\" (UniqueName: \"kubernetes.io/projected/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-kube-api-access-x2lnd\") pod \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\" (UID: \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\") " Nov 02 08:44:44.905724 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:44.905693 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-openshift-ca\") pod \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\" (UID: \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\") " Nov 02 08:44:44.905973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:44.905740 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-xks-ca\") pod \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\" (UID: \"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd\") " Nov 02 08:44:44.906098 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:44.906075 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-openshift-ca" pod "ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd" (UID: "ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:44:44.907915 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:44.907888 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-xks-ca" pod "ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd" (UID: "ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:44:44.908026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:44.907888 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-kube-api-access-x2lnd" pod "ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd" (UID: "ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd"). InnerVolumeSpecName "kube-api-access-x2lnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:44:45.006604 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:45.006564 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:44:45.006604 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:45.006600 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:44:45.006604 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:45.006610 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2lnd\" (UniqueName: \"kubernetes.io/projected/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd-kube-api-access-x2lnd\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:44:45.257568 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:45.257534 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:45.639241 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:45.639145 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" Nov 02 08:44:45.639241 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:45.639192 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5" event={"ID":"ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd","Type":"ContainerDied","Data":"1a86f294d0e3e237205ee17c28970d3a5ef462258f866b5d99eef15cf3f092a7"} Nov 02 08:44:45.639435 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:45.639246 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a86f294d0e3e237205ee17c28970d3a5ef462258f866b5d99eef15cf3f092a7" Nov 02 08:44:47.055645 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:47.055604 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:48.908902 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:48.908855 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:50.701220 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:50.701155 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:52.044482 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:52.044443 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:52.193153 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:52.193121 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:52.533349 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:52.533315 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:54.071233 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:54.071196 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:54.798337 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:54.798306 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:54.910246 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:54.910208 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5"] Nov 02 08:44:54.922421 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:54.922386 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1b623b-p45d5"] Nov 02 08:44:54.966566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:54.966528 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:55.346773 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:55.346736 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:44:55.818056 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:44:55.818018 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd" path="/var/lib/kubelet/pods/ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd/volumes" Nov 02 08:45:00.151686 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.151649 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w"] Nov 02 08:45:00.152092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.152079 5899 cpu_manager.go:414] "containerMap: removing container" podUID="ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd" containerName="revoke-keys" Nov 02 08:45:00.152149 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.152093 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd" containerName="revoke-keys" Nov 02 08:45:00.152149 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.152148 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac9ca014-5a94-44a7-9b0e-e2ad9f24a0bd" containerName="revoke-keys" Nov 02 08:45:00.155547 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.155529 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" Nov 02 08:45:00.173349 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.173325 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:45:00.188412 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.188379 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w"] Nov 02 08:45:00.262484 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.262449 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893485-8nr5w\" (UID: \"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" Nov 02 08:45:00.262484 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.262485 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lslg\" (UniqueName: \"kubernetes.io/projected/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-kube-api-access-5lslg\") pod \"maas-api-key-cleanup-29893485-8nr5w\" (UID: \"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" Nov 02 08:45:00.363959 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.363917 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893485-8nr5w\" (UID: \"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" Nov 02 08:45:00.363959 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.363958 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lslg\" (UniqueName: \"kubernetes.io/projected/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-kube-api-access-5lslg\") pod \"maas-api-key-cleanup-29893485-8nr5w\" (UID: \"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" Nov 02 08:45:00.364650 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.364620 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893485-8nr5w\" (UID: \"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" Nov 02 08:45:00.404113 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.404040 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lslg\" (UniqueName: \"kubernetes.io/projected/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-kube-api-access-5lslg\") pod \"maas-api-key-cleanup-29893485-8nr5w\" (UID: \"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" Nov 02 08:45:00.469243 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.469198 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" Nov 02 08:45:00.773567 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:00.773532 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w"] Nov 02 08:45:00.775929 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:45:00.775899 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e15a99_89cc_4732_8f0a_8e72b23fe4d9.slice/crio-54f6ad831857883309ccee8c8ffe612aa0df74aff0af54d87f838acb613e8652 WatchSource:0}: Error finding container 54f6ad831857883309ccee8c8ffe612aa0df74aff0af54d87f838acb613e8652: Status 404 returned error can't find the container with id 54f6ad831857883309ccee8c8ffe612aa0df74aff0af54d87f838acb613e8652 Nov 02 08:45:01.157815 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:01.157778 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:01.347901 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:01.347828 5899 scope.go:122] "RemoveContainer" containerID="abde0dd204e48f355e66192b290be5cea7e4d25bf1116e1296f2896ca88a2347" Nov 02 08:45:01.723765 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:01.723732 5899 generic.go:356] "Generic (PLEG): container finished" podID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" containerID="57f5674d2a09673aea443f75e5a45ef66034efc175b2bef329ec8b1bcd7abda4" exitCode=60 Nov 02 08:45:01.723963 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:01.723777 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" event={"ID":"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9","Type":"ContainerDied","Data":"57f5674d2a09673aea443f75e5a45ef66034efc175b2bef329ec8b1bcd7abda4"} Nov 02 08:45:01.723963 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:01.723824 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" event={"ID":"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9","Type":"ContainerStarted","Data":"54f6ad831857883309ccee8c8ffe612aa0df74aff0af54d87f838acb613e8652"} Nov 02 08:45:01.724090 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:01.724040 5899 scope.go:122] "RemoveContainer" containerID="57f5674d2a09673aea443f75e5a45ef66034efc175b2bef329ec8b1bcd7abda4" Nov 02 08:45:02.731279 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:02.731245 5899 generic.go:356] "Generic (PLEG): container finished" podID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" containerID="48114e372192b1f7b350f0529b5b4503766e92f9c9cbcdbbb06ea6dfef2aaf72" exitCode=60 Nov 02 08:45:02.731785 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:02.731329 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" event={"ID":"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9","Type":"ContainerDied","Data":"48114e372192b1f7b350f0529b5b4503766e92f9c9cbcdbbb06ea6dfef2aaf72"} Nov 02 08:45:02.731785 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:02.731387 5899 scope.go:122] "RemoveContainer" containerID="57f5674d2a09673aea443f75e5a45ef66034efc175b2bef329ec8b1bcd7abda4" Nov 02 08:45:02.731785 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:02.731650 5899 scope.go:122] "RemoveContainer" containerID="48114e372192b1f7b350f0529b5b4503766e92f9c9cbcdbbb06ea6dfef2aaf72" Nov 02 08:45:02.731979 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:45:02.731956 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893485-8nr5w_odh-ai-gateway-infra(a9e15a99-89cc-4732-8f0a-8e72b23fe4d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" podUID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" Nov 02 08:45:03.738831 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:03.738803 5899 scope.go:122] "RemoveContainer" containerID="48114e372192b1f7b350f0529b5b4503766e92f9c9cbcdbbb06ea6dfef2aaf72" Nov 02 08:45:03.739270 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:45:03.739049 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893485-8nr5w_odh-ai-gateway-infra(a9e15a99-89cc-4732-8f0a-8e72b23fe4d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" podUID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" Nov 02 08:45:06.755364 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:06.755320 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cf7zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:45:06.755927 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:06.755384 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" podUID="b0c571f9-9dc1-493e-b1e5-a256cad13493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:45:11.163245 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:11.163191 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:11.319822 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:11.319792 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:13.140235 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:45:13.140192 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:45:14.812981 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:14.812954 5899 scope.go:122] "RemoveContainer" containerID="48114e372192b1f7b350f0529b5b4503766e92f9c9cbcdbbb06ea6dfef2aaf72" Nov 02 08:45:15.804683 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:15.804643 5899 generic.go:356] "Generic (PLEG): container finished" podID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" containerID="cea7b0b2d3d4e8fd8ff7164313a0bed13e901a68b27307db2b22a75e995493a6" exitCode=60 Nov 02 08:45:15.804854 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:15.804719 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" event={"ID":"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9","Type":"ContainerDied","Data":"cea7b0b2d3d4e8fd8ff7164313a0bed13e901a68b27307db2b22a75e995493a6"} Nov 02 08:45:15.804854 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:15.804793 5899 scope.go:122] "RemoveContainer" containerID="48114e372192b1f7b350f0529b5b4503766e92f9c9cbcdbbb06ea6dfef2aaf72" Nov 02 08:45:15.805154 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:15.805118 5899 scope.go:122] "RemoveContainer" containerID="cea7b0b2d3d4e8fd8ff7164313a0bed13e901a68b27307db2b22a75e995493a6" Nov 02 08:45:15.805423 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:45:15.805403 5899 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893485-8nr5w_odh-ai-gateway-infra(a9e15a99-89cc-4732-8f0a-8e72b23fe4d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" podUID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" Nov 02 08:45:16.541483 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:16.541443 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:16.857555 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:16.857472 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w"] Nov 02 08:45:17.043901 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:17.043871 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" Nov 02 08:45:17.151146 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:17.151061 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-maas-api-service-ca\") pod \"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9\" (UID: \"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9\") " Nov 02 08:45:17.151358 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:17.151222 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-kube-api-access-5lslg\" (UniqueName: \"kubernetes.io/projected/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-kube-api-access-5lslg\") pod \"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9\" (UID: \"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9\") " Nov 02 08:45:17.151530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:17.151502 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-maas-api-service-ca" pod "a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" (UID: "a9e15a99-89cc-4732-8f0a-8e72b23fe4d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:17.153380 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:17.153349 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-kube-api-access-5lslg" pod "a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" (UID: "a9e15a99-89cc-4732-8f0a-8e72b23fe4d9"). InnerVolumeSpecName "kube-api-access-5lslg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:17.252130 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:17.252095 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lslg\" (UniqueName: \"kubernetes.io/projected/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-kube-api-access-5lslg\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:45:17.252130 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:17.252124 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9-maas-api-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:45:17.818822 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:17.818785 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" Nov 02 08:45:17.818822 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:17.818808 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w" event={"ID":"a9e15a99-89cc-4732-8f0a-8e72b23fe4d9","Type":"ContainerDied","Data":"54f6ad831857883309ccee8c8ffe612aa0df74aff0af54d87f838acb613e8652"} Nov 02 08:45:17.819413 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:17.818866 5899 scope.go:122] "RemoveContainer" containerID="cea7b0b2d3d4e8fd8ff7164313a0bed13e901a68b27307db2b22a75e995493a6" Nov 02 08:45:17.884499 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:17.884462 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w"] Nov 02 08:45:17.897268 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:17.897238 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893485-8nr5w"] Nov 02 08:45:19.035054 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:19.035012 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l"] Nov 02 08:45:19.048260 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:19.048224 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-jlw7l"] Nov 02 08:45:19.820958 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:19.820918 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe0319f-9584-4011-973b-fc72d4e171b8" path="/var/lib/kubelet/pods/7fe0319f-9584-4011-973b-fc72d4e171b8/volumes" Nov 02 08:45:19.821510 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:19.821487 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" path="/var/lib/kubelet/pods/a9e15a99-89cc-4732-8f0a-8e72b23fe4d9/volumes" Nov 02 08:45:21.770307 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:21.770264 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cf7zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:45:21.770974 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:21.770938 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" podUID="b0c571f9-9dc1-493e-b1e5-a256cad13493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:45:27.471768 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:27.471719 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:27.622645 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:27.622610 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:33.498001 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.497957 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj"] Nov 02 08:45:33.498604 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.498582 5899 cpu_manager.go:414] "containerMap: removing container" podUID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" containerName="cleanup" Nov 02 08:45:33.498690 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.498607 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" containerName="cleanup" Nov 02 08:45:33.498690 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.498677 5899 cpu_manager.go:414] "containerMap: removing container" podUID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" containerName="cleanup" Nov 02 08:45:33.498804 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.498686 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" containerName="cleanup" Nov 02 08:45:33.498804 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.498704 5899 cpu_manager.go:414] "containerMap: removing container" podUID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" containerName="cleanup" Nov 02 08:45:33.498804 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.498712 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" containerName="cleanup" Nov 02 08:45:33.498968 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.498821 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" containerName="cleanup" Nov 02 08:45:33.498968 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.498833 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" containerName="cleanup" Nov 02 08:45:33.498968 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.498846 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e15a99-89cc-4732-8f0a-8e72b23fe4d9" containerName="cleanup" Nov 02 08:45:33.503400 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.503377 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" Nov 02 08:45:33.519647 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.519618 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:45:33.532757 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.532728 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:45:33.534641 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.534617 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj"] Nov 02 08:45:33.651172 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.651124 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p2slj\" (UID: \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" Nov 02 08:45:33.651365 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.651290 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p2slj\" (UID: \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" Nov 02 08:45:33.651426 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.651407 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxnk5\" (UniqueName: \"kubernetes.io/projected/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-kube-api-access-lxnk5\") pod \"maas-api-revoke-sub-452b6d7b-p2slj\" (UID: \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" Nov 02 08:45:33.752193 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.752061 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p2slj\" (UID: \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" Nov 02 08:45:33.752193 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.752131 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p2slj\" (UID: \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" Nov 02 08:45:33.752454 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.752243 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxnk5\" (UniqueName: \"kubernetes.io/projected/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-kube-api-access-lxnk5\") pod \"maas-api-revoke-sub-452b6d7b-p2slj\" (UID: \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" Nov 02 08:45:33.752891 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.752861 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p2slj\" (UID: \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" Nov 02 08:45:33.754599 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.754573 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p2slj\" (UID: \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" Nov 02 08:45:33.776536 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.776505 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxnk5\" (UniqueName: \"kubernetes.io/projected/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-kube-api-access-lxnk5\") pod \"maas-api-revoke-sub-452b6d7b-p2slj\" (UID: \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" Nov 02 08:45:33.815649 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:33.815620 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" Nov 02 08:45:34.115693 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:34.115584 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj"] Nov 02 08:45:34.118664 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:45:34.118633 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c37fa8_67f8_4884_9b11_2a75bdf6f137.slice/crio-ad9cf767b4a61911539f07873f1661bc113a0fbc29e5b18d354635b3dfc5f008 WatchSource:0}: Error finding container ad9cf767b4a61911539f07873f1661bc113a0fbc29e5b18d354635b3dfc5f008: Status 404 returned error can't find the container with id ad9cf767b4a61911539f07873f1661bc113a0fbc29e5b18d354635b3dfc5f008 Nov 02 08:45:34.911481 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:34.911442 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" event={"ID":"a7c37fa8-67f8-4884-9b11-2a75bdf6f137","Type":"ContainerStarted","Data":"2cff246cc99d5cee1ce712058faa29a55402d4e7931dc7c5ad0482e7e6d42760"} Nov 02 08:45:34.911911 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:34.911496 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" event={"ID":"a7c37fa8-67f8-4884-9b11-2a75bdf6f137","Type":"ContainerStarted","Data":"ad9cf767b4a61911539f07873f1661bc113a0fbc29e5b18d354635b3dfc5f008"} Nov 02 08:45:34.955048 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:34.954986 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" podStartSLOduration=1.954972462 podStartE2EDuration="1.954972462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:45:34.95272991 +0000 UTC m=+4835.251010735" watchObservedRunningTime="2026-11-02 08:45:34.954972462 +0000 UTC m=+4835.253253287" Nov 02 08:45:35.919705 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:35.919662 5899 generic.go:356] "Generic (PLEG): container finished" podID="a7c37fa8-67f8-4884-9b11-2a75bdf6f137" containerID="2cff246cc99d5cee1ce712058faa29a55402d4e7931dc7c5ad0482e7e6d42760" exitCode=0 Nov 02 08:45:35.920258 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:35.919751 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" event={"ID":"a7c37fa8-67f8-4884-9b11-2a75bdf6f137","Type":"ContainerDied","Data":"2cff246cc99d5cee1ce712058faa29a55402d4e7931dc7c5ad0482e7e6d42760"} Nov 02 08:45:37.319702 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:37.319412 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" Nov 02 08:45:37.336270 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:37.335595 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-kube-api-access-lxnk5\" (UniqueName: \"kubernetes.io/projected/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-kube-api-access-lxnk5\") pod \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\" (UID: \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\") " Nov 02 08:45:37.336270 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:37.335690 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-xks-ca\") pod \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\" (UID: \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\") " Nov 02 08:45:37.336270 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:37.335735 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-openshift-ca\") pod \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\" (UID: \"a7c37fa8-67f8-4884-9b11-2a75bdf6f137\") " Nov 02 08:45:37.336679 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:37.336649 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-openshift-ca" pod "a7c37fa8-67f8-4884-9b11-2a75bdf6f137" (UID: "a7c37fa8-67f8-4884-9b11-2a75bdf6f137"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:37.338542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:37.338515 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-xks-ca" pod "a7c37fa8-67f8-4884-9b11-2a75bdf6f137" (UID: "a7c37fa8-67f8-4884-9b11-2a75bdf6f137"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:45:37.338663 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:37.338616 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-kube-api-access-lxnk5" pod "a7c37fa8-67f8-4884-9b11-2a75bdf6f137" (UID: "a7c37fa8-67f8-4884-9b11-2a75bdf6f137"). InnerVolumeSpecName "kube-api-access-lxnk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:37.436888 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:37.436796 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxnk5\" (UniqueName: \"kubernetes.io/projected/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-kube-api-access-lxnk5\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:45:37.436888 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:37.436846 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:45:37.436888 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:37.436861 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c37fa8-67f8-4884-9b11-2a75bdf6f137-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:45:37.937548 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:37.937515 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" Nov 02 08:45:37.937729 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:37.937509 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj" event={"ID":"a7c37fa8-67f8-4884-9b11-2a75bdf6f137","Type":"ContainerDied","Data":"ad9cf767b4a61911539f07873f1661bc113a0fbc29e5b18d354635b3dfc5f008"} Nov 02 08:45:37.937729 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:37.937667 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad9cf767b4a61911539f07873f1661bc113a0fbc29e5b18d354635b3dfc5f008" Nov 02 08:45:38.361205 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:38.359345 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hkc5n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:45:38.361205 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:38.359418 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" podUID="14f48878-440e-45ac-900a-a29238f348c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:45:43.029200 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:43.029154 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65"] Nov 02 08:45:43.042223 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:43.042186 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4ct65"] Nov 02 08:45:43.139710 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:45:43.139674 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:45:43.818748 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:43.818706 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a245ce8f-0097-49ce-869a-d74a1c024296" path="/var/lib/kubelet/pods/a245ce8f-0097-49ce-869a-d74a1c024296/volumes" Nov 02 08:45:44.838278 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:44.838236 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:46.221386 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:46.221351 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:49.779016 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.778983 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj"] Nov 02 08:45:49.779482 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.779466 5899 cpu_manager.go:414] "containerMap: removing container" podUID="a7c37fa8-67f8-4884-9b11-2a75bdf6f137" containerName="revoke-keys" Nov 02 08:45:49.779528 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.779485 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c37fa8-67f8-4884-9b11-2a75bdf6f137" containerName="revoke-keys" Nov 02 08:45:49.779581 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.779571 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c37fa8-67f8-4884-9b11-2a75bdf6f137" containerName="revoke-keys" Nov 02 08:45:49.782946 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.782924 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" Nov 02 08:45:49.800828 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.800776 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:45:49.813956 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.813925 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:45:49.818025 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.817998 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj"] Nov 02 08:45:49.864748 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.864711 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlbr5\" (UniqueName: \"kubernetes.io/projected/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-kube-api-access-qlbr5\") pod \"maas-api-revoke-sub-3e31a506-p2vlj\" (UID: \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" Nov 02 08:45:49.864961 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.864765 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-p2vlj\" (UID: \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" Nov 02 08:45:49.864961 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.864866 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-p2vlj\" (UID: \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" Nov 02 08:45:49.965513 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.965469 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlbr5\" (UniqueName: \"kubernetes.io/projected/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-kube-api-access-qlbr5\") pod \"maas-api-revoke-sub-3e31a506-p2vlj\" (UID: \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" Nov 02 08:45:49.965734 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.965541 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-p2vlj\" (UID: \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" Nov 02 08:45:49.965734 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.965585 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-p2vlj\" (UID: \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" Nov 02 08:45:49.966515 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.966486 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-p2vlj\" (UID: \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" Nov 02 08:45:49.967968 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.967946 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-p2vlj\" (UID: \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" Nov 02 08:45:49.990198 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:49.990148 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlbr5\" (UniqueName: \"kubernetes.io/projected/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-kube-api-access-qlbr5\") pod \"maas-api-revoke-sub-3e31a506-p2vlj\" (UID: \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" Nov 02 08:45:50.095602 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:50.095515 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" Nov 02 08:45:50.401911 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:50.401712 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj"] Nov 02 08:45:50.404943 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:45:50.404906 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54a97ae_1bf3_43b5_9d6b_3a561b0f9ed7.slice/crio-cc5a360931ddd7c665efa23a1c25b138536484f9bcc7fd63c89c509a404e67e1 WatchSource:0}: Error finding container cc5a360931ddd7c665efa23a1c25b138536484f9bcc7fd63c89c509a404e67e1: Status 404 returned error can't find the container with id cc5a360931ddd7c665efa23a1c25b138536484f9bcc7fd63c89c509a404e67e1 Nov 02 08:45:51.006797 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:51.006751 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" event={"ID":"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7","Type":"ContainerStarted","Data":"260c83b42f799a8a9a3d6cfd1593775fb9c94a6e3a8011228d5f156408a9606f"} Nov 02 08:45:51.007337 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:51.006824 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" event={"ID":"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7","Type":"ContainerStarted","Data":"cc5a360931ddd7c665efa23a1c25b138536484f9bcc7fd63c89c509a404e67e1"} Nov 02 08:45:51.051240 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:51.051175 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" podStartSLOduration=2.051139561 podStartE2EDuration="2.051139561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:45:51.048484969 +0000 UTC m=+4851.346765797" watchObservedRunningTime="2026-11-02 08:45:51.051139561 +0000 UTC m=+4851.349420387" Nov 02 08:45:52.034180 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:52.034115 5899 generic.go:356] "Generic (PLEG): container finished" podID="c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7" containerID="260c83b42f799a8a9a3d6cfd1593775fb9c94a6e3a8011228d5f156408a9606f" exitCode=0 Nov 02 08:45:52.034818 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:52.034236 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" event={"ID":"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7","Type":"ContainerDied","Data":"260c83b42f799a8a9a3d6cfd1593775fb9c94a6e3a8011228d5f156408a9606f"} Nov 02 08:45:53.396272 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:53.380274 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hkc5n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:45:53.396272 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:53.380344 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" podUID="14f48878-440e-45ac-900a-a29238f348c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:45:53.439983 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:53.439701 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" Nov 02 08:45:53.540523 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:53.540477 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-kube-api-access-qlbr5\" (UniqueName: \"kubernetes.io/projected/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-kube-api-access-qlbr5\") pod \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\" (UID: \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\") " Nov 02 08:45:53.540730 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:53.540564 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-xks-ca\") pod \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\" (UID: \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\") " Nov 02 08:45:53.540730 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:53.540697 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-openshift-ca\") pod \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\" (UID: \"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7\") " Nov 02 08:45:53.541282 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:53.541254 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-openshift-ca" pod "c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7" (UID: "c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:45:53.545466 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:53.543384 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-kube-api-access-qlbr5" pod "c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7" (UID: "c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7"). InnerVolumeSpecName "kube-api-access-qlbr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:45:53.545466 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:53.545835 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-xks-ca" pod "c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7" (UID: "c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:45:53.641706 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:53.641663 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:45:53.641706 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:53.641698 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlbr5\" (UniqueName: \"kubernetes.io/projected/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-kube-api-access-qlbr5\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:45:53.641706 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:53.641710 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:45:54.051082 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:54.051048 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" Nov 02 08:45:54.051271 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:54.051081 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj" event={"ID":"c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7","Type":"ContainerDied","Data":"cc5a360931ddd7c665efa23a1c25b138536484f9bcc7fd63c89c509a404e67e1"} Nov 02 08:45:54.051271 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:54.051134 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc5a360931ddd7c665efa23a1c25b138536484f9bcc7fd63c89c509a404e67e1" Nov 02 08:45:57.370717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:57.370678 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:57.517590 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:57.517553 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:45:59.070863 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:59.070819 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx"] Nov 02 08:45:59.082057 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:59.082028 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vgsbx"] Nov 02 08:45:59.818277 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:45:59.818245 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cabc0e3-6e2e-4b1f-a9b7-76db16312d68" path="/var/lib/kubelet/pods/7cabc0e3-6e2e-4b1f-a9b7-76db16312d68/volumes" Nov 02 08:46:01.373071 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:01.373036 5899 scope.go:122] "RemoveContainer" containerID="85464df1f8a4a33133514c4cb6418ac0e3af4903c9dfc31ec1ab7c5554d58dbb" Nov 02 08:46:08.351408 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:08.351351 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hkc5n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:46:08.351889 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:08.351424 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hkc5n" podUID="14f48878-440e-45ac-900a-a29238f348c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:46:09.772544 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:09.772511 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw"] Nov 02 08:46:09.772974 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:09.772961 5899 cpu_manager.go:414] "containerMap: removing container" podUID="c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7" containerName="revoke-keys" Nov 02 08:46:09.773019 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:09.772975 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7" containerName="revoke-keys" Nov 02 08:46:09.773055 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:09.773031 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7" containerName="revoke-keys" Nov 02 08:46:09.777529 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:09.777509 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" Nov 02 08:46:09.796001 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:09.795973 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:46:09.809359 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:09.809331 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:46:09.811077 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:09.811055 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw"] Nov 02 08:46:09.923819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:09.923782 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fc45422-gqxbw\" (UID: \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" Nov 02 08:46:09.924031 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:09.923831 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjz2\" (UniqueName: \"kubernetes.io/projected/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-kube-api-access-nnjz2\") pod \"maas-api-revoke-sub-3fc45422-gqxbw\" (UID: \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" Nov 02 08:46:09.924031 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:09.923902 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fc45422-gqxbw\" (UID: \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" Nov 02 08:46:10.025178 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:10.025066 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fc45422-gqxbw\" (UID: \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" Nov 02 08:46:10.025178 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:10.025117 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjz2\" (UniqueName: \"kubernetes.io/projected/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-kube-api-access-nnjz2\") pod \"maas-api-revoke-sub-3fc45422-gqxbw\" (UID: \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" Nov 02 08:46:10.025178 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:10.025148 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fc45422-gqxbw\" (UID: \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" Nov 02 08:46:10.025785 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:10.025751 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fc45422-gqxbw\" (UID: \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" Nov 02 08:46:10.027543 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:10.027521 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fc45422-gqxbw\" (UID: \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" Nov 02 08:46:10.033269 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:10.033235 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8"] Nov 02 08:46:10.037898 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:10.037878 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5zcw8"] Nov 02 08:46:10.051780 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:10.051753 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjz2\" (UniqueName: \"kubernetes.io/projected/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-kube-api-access-nnjz2\") pod \"maas-api-revoke-sub-3fc45422-gqxbw\" (UID: \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" Nov 02 08:46:10.090389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:10.090354 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" Nov 02 08:46:10.393071 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:10.393043 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw"] Nov 02 08:46:10.395801 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:46:10.395773 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6f7114_c90d_4fc9_96a7_c955fd1c87aa.slice/crio-b813204fb7dd97447fdefeb07eda2bd8dd53526f0f210f2136fdc48f989e3aaf WatchSource:0}: Error finding container b813204fb7dd97447fdefeb07eda2bd8dd53526f0f210f2136fdc48f989e3aaf: Status 404 returned error can't find the container with id b813204fb7dd97447fdefeb07eda2bd8dd53526f0f210f2136fdc48f989e3aaf Nov 02 08:46:11.159092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:11.159044 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" event={"ID":"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa","Type":"ContainerStarted","Data":"bdd8388894faf0ec9dd6f1b75a0409a1c62f183047a891959089fccf45ffae79"} Nov 02 08:46:11.159508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:11.159116 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" event={"ID":"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa","Type":"ContainerStarted","Data":"b813204fb7dd97447fdefeb07eda2bd8dd53526f0f210f2136fdc48f989e3aaf"} Nov 02 08:46:11.202627 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:11.202573 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" podStartSLOduration=2.202556588 podStartE2EDuration="2.202556588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:46:11.201484121 +0000 UTC m=+4871.499764961" watchObservedRunningTime="2026-11-02 08:46:11.202556588 +0000 UTC m=+4871.500837413" Nov 02 08:46:11.309003 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:11.308971 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:46:11.819688 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:11.819655 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab28fc7-3145-4597-87ec-8284ed8d507a" path="/var/lib/kubelet/pods/aab28fc7-3145-4597-87ec-8284ed8d507a/volumes" Nov 02 08:46:12.165925 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:12.165838 5899 generic.go:356] "Generic (PLEG): container finished" podID="cd6f7114-c90d-4fc9-96a7-c955fd1c87aa" containerID="bdd8388894faf0ec9dd6f1b75a0409a1c62f183047a891959089fccf45ffae79" exitCode=0 Nov 02 08:46:12.165925 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:12.165891 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" event={"ID":"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa","Type":"ContainerDied","Data":"bdd8388894faf0ec9dd6f1b75a0409a1c62f183047a891959089fccf45ffae79"} Nov 02 08:46:13.140825 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:46:13.140785 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:46:13.318727 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:13.318700 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" Nov 02 08:46:13.457765 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:13.457677 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-xks-ca\") pod \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\" (UID: \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\") " Nov 02 08:46:13.457765 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:13.457730 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-openshift-ca\") pod \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\" (UID: \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\") " Nov 02 08:46:13.457765 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:13.457764 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-kube-api-access-nnjz2\" (UniqueName: \"kubernetes.io/projected/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-kube-api-access-nnjz2\") pod \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\" (UID: \"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa\") " Nov 02 08:46:13.458122 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:13.458095 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-openshift-ca" pod "cd6f7114-c90d-4fc9-96a7-c955fd1c87aa" (UID: "cd6f7114-c90d-4fc9-96a7-c955fd1c87aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:46:13.459854 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:13.459829 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-kube-api-access-nnjz2" pod "cd6f7114-c90d-4fc9-96a7-c955fd1c87aa" (UID: "cd6f7114-c90d-4fc9-96a7-c955fd1c87aa"). InnerVolumeSpecName "kube-api-access-nnjz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:46:13.459854 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:13.459844 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-xks-ca" pod "cd6f7114-c90d-4fc9-96a7-c955fd1c87aa" (UID: "cd6f7114-c90d-4fc9-96a7-c955fd1c87aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:46:13.559105 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:13.559065 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:46:13.559105 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:13.559097 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:46:13.559105 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:13.559107 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnjz2\" (UniqueName: \"kubernetes.io/projected/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa-kube-api-access-nnjz2\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:46:14.176766 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:14.176693 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" Nov 02 08:46:14.176766 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:14.176724 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw" event={"ID":"cd6f7114-c90d-4fc9-96a7-c955fd1c87aa","Type":"ContainerDied","Data":"b813204fb7dd97447fdefeb07eda2bd8dd53526f0f210f2136fdc48f989e3aaf"} Nov 02 08:46:14.176950 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:14.176772 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b813204fb7dd97447fdefeb07eda2bd8dd53526f0f210f2136fdc48f989e3aaf" Nov 02 08:46:43.140615 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:46:43.140571 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:46:48.402345 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:46:48.402309 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:01.402389 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:01.402353 5899 scope.go:122] "RemoveContainer" containerID="851dadfdad965d455e9a1a3ad585d14003908875bd38b34efb43076e6e139e0d" Nov 02 08:47:01.411154 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:01.411131 5899 scope.go:122] "RemoveContainer" containerID="7ccfd2b661dc4b102ae3842c5cc4c2ee6ac51e2419dcbff57f66beee364ef1fb" Nov 02 08:47:01.419588 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:01.419564 5899 scope.go:122] "RemoveContainer" containerID="4ab4d9642582a50bdd45312f7fa0243eb4aa9dc99ea630e90ad320da537ddff7" Nov 02 08:47:07.567911 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:07.567866 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:07.762826 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:07.762785 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:08.564256 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:08.564217 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:09.853427 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:09.853392 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:11.423247 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:11.423130 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:47:13.032332 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:13.032198 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj"] Nov 02 08:47:13.043765 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:13.043727 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce062c03-lvtgj"] Nov 02 08:47:13.146280 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:47:13.146196 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:47:13.818241 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:13.818197 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38228fca-dfc7-4041-ae75-5099d9946e8b" path="/var/lib/kubelet/pods/38228fca-dfc7-4041-ae75-5099d9946e8b/volumes" Nov 02 08:47:28.037986 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:28.037939 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz"] Nov 02 08:47:28.051146 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:28.051101 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b72af5-m4tzz"] Nov 02 08:47:29.817919 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:47:29.817880 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29171543-639c-4da8-ba3a-56656f8a9f9d" path="/var/lib/kubelet/pods/29171543-639c-4da8-ba3a-56656f8a9f9d/volumes" Nov 02 08:47:43.143125 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:47:43.143044 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:48:01.448874 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:01.448822 5899 scope.go:122] "RemoveContainer" containerID="b228f8ce43da83572ed37701c04dd488d82d5669580d9f938dab9418e08fe563" Nov 02 08:48:12.623594 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:12.623553 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:13.142746 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:48:13.142700 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:48:29.962651 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:29.962617 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:37.316566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.316528 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp"] Nov 02 08:48:37.316981 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.316940 5899 cpu_manager.go:414] "containerMap: removing container" podUID="cd6f7114-c90d-4fc9-96a7-c955fd1c87aa" containerName="revoke-keys" Nov 02 08:48:37.316981 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.316950 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6f7114-c90d-4fc9-96a7-c955fd1c87aa" containerName="revoke-keys" Nov 02 08:48:37.317069 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.317033 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6f7114-c90d-4fc9-96a7-c955fd1c87aa" containerName="revoke-keys" Nov 02 08:48:37.320284 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.320266 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" Nov 02 08:48:37.337953 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.337928 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:48:37.351104 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.351064 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:48:37.352894 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.352867 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp"] Nov 02 08:48:37.367486 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.367460 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-s2dzp\" (UID: \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" Nov 02 08:48:37.367601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.367585 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-s2dzp\" (UID: \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" Nov 02 08:48:37.367645 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.367614 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fjvw\" (UniqueName: \"kubernetes.io/projected/d2ec716e-c29f-431b-a58d-1f31cb759ea5-kube-api-access-5fjvw\") pod \"maas-api-revoke-sub-e2547622-s2dzp\" (UID: \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" Nov 02 08:48:37.468673 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.468633 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-s2dzp\" (UID: \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" Nov 02 08:48:37.468673 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.468680 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fjvw\" (UniqueName: \"kubernetes.io/projected/d2ec716e-c29f-431b-a58d-1f31cb759ea5-kube-api-access-5fjvw\") pod \"maas-api-revoke-sub-e2547622-s2dzp\" (UID: \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" Nov 02 08:48:37.468888 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.468717 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-s2dzp\" (UID: \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" Nov 02 08:48:37.469450 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.469423 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-s2dzp\" (UID: \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" Nov 02 08:48:37.471024 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.471000 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-s2dzp\" (UID: \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" Nov 02 08:48:37.493221 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.493196 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fjvw\" (UniqueName: \"kubernetes.io/projected/d2ec716e-c29f-431b-a58d-1f31cb759ea5-kube-api-access-5fjvw\") pod \"maas-api-revoke-sub-e2547622-s2dzp\" (UID: \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" Nov 02 08:48:37.633283 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.633189 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" Nov 02 08:48:37.936150 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.936118 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp"] Nov 02 08:48:37.937216 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:48:37.937190 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ec716e_c29f_431b_a58d_1f31cb759ea5.slice/crio-192646db802388b5e5bfe4703bb8df2f6cd15df32baf042b852dddda89944c19 WatchSource:0}: Error finding container 192646db802388b5e5bfe4703bb8df2f6cd15df32baf042b852dddda89944c19: Status 404 returned error can't find the container with id 192646db802388b5e5bfe4703bb8df2f6cd15df32baf042b852dddda89944c19 Nov 02 08:48:37.973238 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:37.973206 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" event={"ID":"d2ec716e-c29f-431b-a58d-1f31cb759ea5","Type":"ContainerStarted","Data":"192646db802388b5e5bfe4703bb8df2f6cd15df32baf042b852dddda89944c19"} Nov 02 08:48:38.979771 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:38.979730 5899 generic.go:356] "Generic (PLEG): container finished" podID="d2ec716e-c29f-431b-a58d-1f31cb759ea5" containerID="e8a2b2f8c17985330cac81604428962ce9b8547e31271a4b6d080dfdc8b49ce6" exitCode=0 Nov 02 08:48:38.980193 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:38.979809 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" event={"ID":"d2ec716e-c29f-431b-a58d-1f31cb759ea5","Type":"ContainerDied","Data":"e8a2b2f8c17985330cac81604428962ce9b8547e31271a4b6d080dfdc8b49ce6"} Nov 02 08:48:40.129671 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:40.129648 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" Nov 02 08:48:40.192136 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:40.192104 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ec716e-c29f-431b-a58d-1f31cb759ea5-kube-api-access-5fjvw\" (UniqueName: \"kubernetes.io/projected/d2ec716e-c29f-431b-a58d-1f31cb759ea5-kube-api-access-5fjvw\") pod \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\" (UID: \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\") " Nov 02 08:48:40.192368 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:40.192237 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-xks-ca\") pod \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\" (UID: \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\") " Nov 02 08:48:40.192368 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:40.192329 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-openshift-ca\") pod \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\" (UID: \"d2ec716e-c29f-431b-a58d-1f31cb759ea5\") " Nov 02 08:48:40.192738 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:40.192711 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-openshift-ca" pod "d2ec716e-c29f-431b-a58d-1f31cb759ea5" (UID: "d2ec716e-c29f-431b-a58d-1f31cb759ea5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:48:40.194351 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:40.194325 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ec716e-c29f-431b-a58d-1f31cb759ea5-kube-api-access-5fjvw" pod "d2ec716e-c29f-431b-a58d-1f31cb759ea5" (UID: "d2ec716e-c29f-431b-a58d-1f31cb759ea5"). InnerVolumeSpecName "kube-api-access-5fjvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:48:40.194711 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:40.194694 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-xks-ca" pod "d2ec716e-c29f-431b-a58d-1f31cb759ea5" (UID: "d2ec716e-c29f-431b-a58d-1f31cb759ea5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:48:40.293679 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:40.293588 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:48:40.293679 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:40.293620 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fjvw\" (UniqueName: \"kubernetes.io/projected/d2ec716e-c29f-431b-a58d-1f31cb759ea5-kube-api-access-5fjvw\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:48:40.293679 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:40.293630 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ec716e-c29f-431b-a58d-1f31cb759ea5-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:48:40.992142 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:40.992113 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" Nov 02 08:48:40.992344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:40.992113 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp" event={"ID":"d2ec716e-c29f-431b-a58d-1f31cb759ea5","Type":"ContainerDied","Data":"192646db802388b5e5bfe4703bb8df2f6cd15df32baf042b852dddda89944c19"} Nov 02 08:48:40.992344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:40.992229 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="192646db802388b5e5bfe4703bb8df2f6cd15df32baf042b852dddda89944c19" Nov 02 08:48:43.140399 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:48:43.140356 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:48:48.227256 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:48.227215 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:48.368852 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:48.368813 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:48.518329 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:48.518243 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:48:58.967185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:58.964278 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7xqxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:48:58.967185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:48:58.964383 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" podUID="e43d749b-223f-4a0a-997a-a0637ddf2d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 08:49:00.697740 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.697703 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s"] Nov 02 08:49:00.698128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.698108 5899 cpu_manager.go:414] "containerMap: removing container" podUID="d2ec716e-c29f-431b-a58d-1f31cb759ea5" containerName="revoke-keys" Nov 02 08:49:00.698128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.698119 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ec716e-c29f-431b-a58d-1f31cb759ea5" containerName="revoke-keys" Nov 02 08:49:00.698263 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.698228 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ec716e-c29f-431b-a58d-1f31cb759ea5" containerName="revoke-keys" Nov 02 08:49:00.701442 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.701424 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" Nov 02 08:49:00.732805 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.732774 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:49:00.732805 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.732807 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:49:00.734703 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.734668 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s"] Nov 02 08:49:00.750063 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.750033 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9hl7s\" (UID: \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" Nov 02 08:49:00.750222 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.750083 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vps49\" (UniqueName: \"kubernetes.io/projected/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-kube-api-access-vps49\") pod \"maas-api-revoke-sub-c1e271a9-9hl7s\" (UID: \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" Nov 02 08:49:00.750222 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.750207 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9hl7s\" (UID: \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" Nov 02 08:49:00.851338 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.851299 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9hl7s\" (UID: \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" Nov 02 08:49:00.851675 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.851360 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vps49\" (UniqueName: \"kubernetes.io/projected/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-kube-api-access-vps49\") pod \"maas-api-revoke-sub-c1e271a9-9hl7s\" (UID: \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" Nov 02 08:49:00.851675 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.851415 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9hl7s\" (UID: \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" Nov 02 08:49:00.852051 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.852023 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9hl7s\" (UID: \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" Nov 02 08:49:00.853855 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.853836 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9hl7s\" (UID: \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" Nov 02 08:49:00.875675 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:00.875642 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vps49\" (UniqueName: \"kubernetes.io/projected/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-kube-api-access-vps49\") pod \"maas-api-revoke-sub-c1e271a9-9hl7s\" (UID: \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" Nov 02 08:49:01.014595 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:01.014558 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" Nov 02 08:49:01.316253 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:01.316224 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s"] Nov 02 08:49:01.318178 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:49:01.318137 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4bfd0e_bd84_4bc8_af11_4c8602afa009.slice/crio-77b32e0ef1590446ed122cd16d942ad5d198d122a51c13422d773cdb327ddb74 WatchSource:0}: Error finding container 77b32e0ef1590446ed122cd16d942ad5d198d122a51c13422d773cdb327ddb74: Status 404 returned error can't find the container with id 77b32e0ef1590446ed122cd16d942ad5d198d122a51c13422d773cdb327ddb74 Nov 02 08:49:01.472742 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:01.472715 5899 scope.go:122] "RemoveContainer" containerID="ddd70a24ed6b8fa89e822255d3b7a2eaa105314c23e34ebd66592dd228b6e5ec" Nov 02 08:49:01.481446 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:01.481422 5899 scope.go:122] "RemoveContainer" containerID="f6eaa7c52b395e758f685a1fe46833f145f0e20e6ed7771893b3c1f49b0700f2" Nov 02 08:49:02.128780 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:02.128744 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" event={"ID":"1c4bfd0e-bd84-4bc8-af11-4c8602afa009","Type":"ContainerStarted","Data":"1c273cbb68cac137f3392116bed6c39d163bae65b02693f45f58f94307733869"} Nov 02 08:49:02.129213 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:02.128803 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" event={"ID":"1c4bfd0e-bd84-4bc8-af11-4c8602afa009","Type":"ContainerStarted","Data":"77b32e0ef1590446ed122cd16d942ad5d198d122a51c13422d773cdb327ddb74"} Nov 02 08:49:02.172350 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:02.172296 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" podStartSLOduration=2.172278448 podStartE2EDuration="2.172278448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:49:02.171149048 +0000 UTC m=+5042.469429872" watchObservedRunningTime="2026-11-02 08:49:02.172278448 +0000 UTC m=+5042.470559273" Nov 02 08:49:03.135582 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:03.135546 5899 generic.go:356] "Generic (PLEG): container finished" podID="1c4bfd0e-bd84-4bc8-af11-4c8602afa009" containerID="1c273cbb68cac137f3392116bed6c39d163bae65b02693f45f58f94307733869" exitCode=0 Nov 02 08:49:03.136025 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:03.135639 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" event={"ID":"1c4bfd0e-bd84-4bc8-af11-4c8602afa009","Type":"ContainerDied","Data":"1c273cbb68cac137f3392116bed6c39d163bae65b02693f45f58f94307733869"} Nov 02 08:49:04.277453 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:04.277420 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" Nov 02 08:49:04.386860 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:04.386824 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-openshift-ca\") pod \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\" (UID: \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\") " Nov 02 08:49:04.387054 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:04.387002 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-kube-api-access-vps49\" (UniqueName: \"kubernetes.io/projected/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-kube-api-access-vps49\") pod \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\" (UID: \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\") " Nov 02 08:49:04.387054 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:04.387039 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-xks-ca\") pod \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\" (UID: \"1c4bfd0e-bd84-4bc8-af11-4c8602afa009\") " Nov 02 08:49:04.387276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:04.387253 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-openshift-ca" pod "1c4bfd0e-bd84-4bc8-af11-4c8602afa009" (UID: "1c4bfd0e-bd84-4bc8-af11-4c8602afa009"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:49:04.387365 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:04.387353 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:49:04.389204 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:04.389154 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-kube-api-access-vps49" pod "1c4bfd0e-bd84-4bc8-af11-4c8602afa009" (UID: "1c4bfd0e-bd84-4bc8-af11-4c8602afa009"). InnerVolumeSpecName "kube-api-access-vps49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:04.389204 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:04.389169 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-xks-ca" pod "1c4bfd0e-bd84-4bc8-af11-4c8602afa009" (UID: "1c4bfd0e-bd84-4bc8-af11-4c8602afa009"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:49:04.488666 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:04.488628 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vps49\" (UniqueName: \"kubernetes.io/projected/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-kube-api-access-vps49\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:49:04.488666 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:04.488662 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4bfd0e-bd84-4bc8-af11-4c8602afa009-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:49:05.147219 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:05.147184 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" Nov 02 08:49:05.147219 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:05.147213 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s" event={"ID":"1c4bfd0e-bd84-4bc8-af11-4c8602afa009","Type":"ContainerDied","Data":"77b32e0ef1590446ed122cd16d942ad5d198d122a51c13422d773cdb327ddb74"} Nov 02 08:49:05.147491 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:05.147263 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77b32e0ef1590446ed122cd16d942ad5d198d122a51c13422d773cdb327ddb74" Nov 02 08:49:11.651716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:11.651590 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:49:13.141012 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:49:13.140965 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:49:20.708544 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.708511 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc"] Nov 02 08:49:20.709005 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.708919 5899 cpu_manager.go:414] "containerMap: removing container" podUID="1c4bfd0e-bd84-4bc8-af11-4c8602afa009" containerName="revoke-keys" Nov 02 08:49:20.709005 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.708930 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4bfd0e-bd84-4bc8-af11-4c8602afa009" containerName="revoke-keys" Nov 02 08:49:20.709087 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.709006 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4bfd0e-bd84-4bc8-af11-4c8602afa009" containerName="revoke-keys" Nov 02 08:49:20.712474 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.712454 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" Nov 02 08:49:20.733591 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.733553 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc"] Nov 02 08:49:20.743047 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.743015 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:49:20.743218 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.743071 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:49:20.861930 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.861893 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6v2\" (UniqueName: \"kubernetes.io/projected/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-kube-api-access-ck6v2\") pod \"maas-api-revoke-sub-2e1f5b5f-wl4xc\" (UID: \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" Nov 02 08:49:20.862112 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.861948 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wl4xc\" (UID: \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" Nov 02 08:49:20.862112 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.862067 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wl4xc\" (UID: \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" Nov 02 08:49:20.963521 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.963439 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6v2\" (UniqueName: \"kubernetes.io/projected/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-kube-api-access-ck6v2\") pod \"maas-api-revoke-sub-2e1f5b5f-wl4xc\" (UID: \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" Nov 02 08:49:20.963521 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.963498 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wl4xc\" (UID: \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" Nov 02 08:49:20.963745 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.963549 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wl4xc\" (UID: \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" Nov 02 08:49:20.964288 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.964266 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wl4xc\" (UID: \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" Nov 02 08:49:20.965922 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.965896 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wl4xc\" (UID: \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" Nov 02 08:49:20.987726 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:20.987687 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6v2\" (UniqueName: \"kubernetes.io/projected/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-kube-api-access-ck6v2\") pod \"maas-api-revoke-sub-2e1f5b5f-wl4xc\" (UID: \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" Nov 02 08:49:21.028717 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:21.028682 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" Nov 02 08:49:21.327313 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:21.327257 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc"] Nov 02 08:49:21.330307 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:49:21.330277 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6a5c24_c1c8_42e0_9638_53c9c7f787f0.slice/crio-98c1bdb531f52c04095d5e7819912c5d6048f8dbb3a75a3a95b6b18627260d7c WatchSource:0}: Error finding container 98c1bdb531f52c04095d5e7819912c5d6048f8dbb3a75a3a95b6b18627260d7c: Status 404 returned error can't find the container with id 98c1bdb531f52c04095d5e7819912c5d6048f8dbb3a75a3a95b6b18627260d7c Nov 02 08:49:22.253601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:22.253569 5899 generic.go:356] "Generic (PLEG): container finished" podID="2a6a5c24-c1c8-42e0-9638-53c9c7f787f0" containerID="277ab3515aed2d276036ec6beff5a19d1a4c77ba06e594d6b60025e1c1cab480" exitCode=0 Nov 02 08:49:22.254004 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:22.253640 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" event={"ID":"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0","Type":"ContainerDied","Data":"277ab3515aed2d276036ec6beff5a19d1a4c77ba06e594d6b60025e1c1cab480"} Nov 02 08:49:22.254004 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:22.253676 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" event={"ID":"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0","Type":"ContainerStarted","Data":"98c1bdb531f52c04095d5e7819912c5d6048f8dbb3a75a3a95b6b18627260d7c"} Nov 02 08:49:23.396209 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:23.396177 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" Nov 02 08:49:23.488481 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:23.488443 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-kube-api-access-ck6v2\" (UniqueName: \"kubernetes.io/projected/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-kube-api-access-ck6v2\") pod \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\" (UID: \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\") " Nov 02 08:49:23.488682 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:23.488535 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-xks-ca\") pod \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\" (UID: \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\") " Nov 02 08:49:23.488682 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:23.488588 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-openshift-ca\") pod \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\" (UID: \"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0\") " Nov 02 08:49:23.488971 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:23.488945 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-openshift-ca" pod "2a6a5c24-c1c8-42e0-9638-53c9c7f787f0" (UID: "2a6a5c24-c1c8-42e0-9638-53c9c7f787f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:49:23.490570 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:23.490544 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-xks-ca" pod "2a6a5c24-c1c8-42e0-9638-53c9c7f787f0" (UID: "2a6a5c24-c1c8-42e0-9638-53c9c7f787f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:49:23.490685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:23.490546 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-kube-api-access-ck6v2" pod "2a6a5c24-c1c8-42e0-9638-53c9c7f787f0" (UID: "2a6a5c24-c1c8-42e0-9638-53c9c7f787f0"). InnerVolumeSpecName "kube-api-access-ck6v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:23.590027 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:23.589943 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6v2\" (UniqueName: \"kubernetes.io/projected/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-kube-api-access-ck6v2\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:49:23.590027 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:23.589979 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:49:23.590027 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:23.589990 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:49:24.265114 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:24.265082 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" Nov 02 08:49:24.265327 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:24.265113 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc" event={"ID":"2a6a5c24-c1c8-42e0-9638-53c9c7f787f0","Type":"ContainerDied","Data":"98c1bdb531f52c04095d5e7819912c5d6048f8dbb3a75a3a95b6b18627260d7c"} Nov 02 08:49:24.265327 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:24.265183 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98c1bdb531f52c04095d5e7819912c5d6048f8dbb3a75a3a95b6b18627260d7c" Nov 02 08:49:32.002689 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.002655 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq"] Nov 02 08:49:32.003092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.003073 5899 cpu_manager.go:414] "containerMap: removing container" podUID="2a6a5c24-c1c8-42e0-9638-53c9c7f787f0" containerName="revoke-keys" Nov 02 08:49:32.003092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.003084 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a6a5c24-c1c8-42e0-9638-53c9c7f787f0" containerName="revoke-keys" Nov 02 08:49:32.003197 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.003187 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a6a5c24-c1c8-42e0-9638-53c9c7f787f0" containerName="revoke-keys" Nov 02 08:49:32.006462 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.006443 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" Nov 02 08:49:32.023941 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.023910 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:49:32.037270 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.037231 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:49:32.039127 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.039095 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq"] Nov 02 08:49:32.172395 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.172359 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b92vw\" (UniqueName: \"kubernetes.io/projected/028899e5-16ed-4a11-b932-f5264ce2bae9-kube-api-access-b92vw\") pod \"maas-api-revoke-sub-ec843018-24nxq\" (UID: \"028899e5-16ed-4a11-b932-f5264ce2bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" Nov 02 08:49:32.172574 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.172432 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-24nxq\" (UID: \"028899e5-16ed-4a11-b932-f5264ce2bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" Nov 02 08:49:32.172574 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.172539 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-24nxq\" (UID: \"028899e5-16ed-4a11-b932-f5264ce2bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" Nov 02 08:49:32.273526 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.273437 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-24nxq\" (UID: \"028899e5-16ed-4a11-b932-f5264ce2bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" Nov 02 08:49:32.273526 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.273510 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-24nxq\" (UID: \"028899e5-16ed-4a11-b932-f5264ce2bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" Nov 02 08:49:32.273729 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.273552 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b92vw\" (UniqueName: \"kubernetes.io/projected/028899e5-16ed-4a11-b932-f5264ce2bae9-kube-api-access-b92vw\") pod \"maas-api-revoke-sub-ec843018-24nxq\" (UID: \"028899e5-16ed-4a11-b932-f5264ce2bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" Nov 02 08:49:32.274128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.274100 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-24nxq\" (UID: \"028899e5-16ed-4a11-b932-f5264ce2bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" Nov 02 08:49:32.275885 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.275857 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-24nxq\" (UID: \"028899e5-16ed-4a11-b932-f5264ce2bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" Nov 02 08:49:32.297543 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.297521 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b92vw\" (UniqueName: \"kubernetes.io/projected/028899e5-16ed-4a11-b932-f5264ce2bae9-kube-api-access-b92vw\") pod \"maas-api-revoke-sub-ec843018-24nxq\" (UID: \"028899e5-16ed-4a11-b932-f5264ce2bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" Nov 02 08:49:32.327575 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.327551 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" Nov 02 08:49:32.631542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:32.631514 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq"] Nov 02 08:49:32.633804 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:49:32.633775 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028899e5_16ed_4a11_b932_f5264ce2bae9.slice/crio-c428a2386b8f306b1e04df5009a92c9cc6a75942328e8ac1594751326b072f3f WatchSource:0}: Error finding container c428a2386b8f306b1e04df5009a92c9cc6a75942328e8ac1594751326b072f3f: Status 404 returned error can't find the container with id c428a2386b8f306b1e04df5009a92c9cc6a75942328e8ac1594751326b072f3f Nov 02 08:49:33.312101 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:33.312061 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" event={"ID":"028899e5-16ed-4a11-b932-f5264ce2bae9","Type":"ContainerStarted","Data":"79a62cc84b57350d1b511fad54191076b7a7c36e2a58e09bd16e1d0264c18da1"} Nov 02 08:49:33.312532 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:33.312121 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" event={"ID":"028899e5-16ed-4a11-b932-f5264ce2bae9","Type":"ContainerStarted","Data":"c428a2386b8f306b1e04df5009a92c9cc6a75942328e8ac1594751326b072f3f"} Nov 02 08:49:33.356075 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:33.356019 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" podStartSLOduration=2.356002584 podStartE2EDuration="2.356002584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:49:33.353765333 +0000 UTC m=+5073.652046157" watchObservedRunningTime="2026-11-02 08:49:33.356002584 +0000 UTC m=+5073.654283431" Nov 02 08:49:34.319586 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:34.319545 5899 generic.go:356] "Generic (PLEG): container finished" podID="028899e5-16ed-4a11-b932-f5264ce2bae9" containerID="79a62cc84b57350d1b511fad54191076b7a7c36e2a58e09bd16e1d0264c18da1" exitCode=0 Nov 02 08:49:34.319990 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:34.319631 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" event={"ID":"028899e5-16ed-4a11-b932-f5264ce2bae9","Type":"ContainerDied","Data":"79a62cc84b57350d1b511fad54191076b7a7c36e2a58e09bd16e1d0264c18da1"} Nov 02 08:49:35.453198 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:35.453156 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" Nov 02 08:49:35.610380 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:35.610286 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-openshift-ca\") pod \"028899e5-16ed-4a11-b932-f5264ce2bae9\" (UID: \"028899e5-16ed-4a11-b932-f5264ce2bae9\") " Nov 02 08:49:35.610562 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:35.610383 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028899e5-16ed-4a11-b932-f5264ce2bae9-kube-api-access-b92vw\" (UniqueName: \"kubernetes.io/projected/028899e5-16ed-4a11-b932-f5264ce2bae9-kube-api-access-b92vw\") pod \"028899e5-16ed-4a11-b932-f5264ce2bae9\" (UID: \"028899e5-16ed-4a11-b932-f5264ce2bae9\") " Nov 02 08:49:35.610562 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:35.610481 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-xks-ca\") pod \"028899e5-16ed-4a11-b932-f5264ce2bae9\" (UID: \"028899e5-16ed-4a11-b932-f5264ce2bae9\") " Nov 02 08:49:35.610680 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:35.610639 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-openshift-ca" pod "028899e5-16ed-4a11-b932-f5264ce2bae9" (UID: "028899e5-16ed-4a11-b932-f5264ce2bae9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:49:35.610822 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:35.610802 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:49:35.612558 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:35.612535 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-xks-ca" pod "028899e5-16ed-4a11-b932-f5264ce2bae9" (UID: "028899e5-16ed-4a11-b932-f5264ce2bae9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:49:35.612643 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:35.612556 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028899e5-16ed-4a11-b932-f5264ce2bae9-kube-api-access-b92vw" pod "028899e5-16ed-4a11-b932-f5264ce2bae9" (UID: "028899e5-16ed-4a11-b932-f5264ce2bae9"). InnerVolumeSpecName "kube-api-access-b92vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:49:35.711807 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:35.711776 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028899e5-16ed-4a11-b932-f5264ce2bae9-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:49:35.711807 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:35.711804 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b92vw\" (UniqueName: \"kubernetes.io/projected/028899e5-16ed-4a11-b932-f5264ce2bae9-kube-api-access-b92vw\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:49:36.331651 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:36.331613 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" event={"ID":"028899e5-16ed-4a11-b932-f5264ce2bae9","Type":"ContainerDied","Data":"c428a2386b8f306b1e04df5009a92c9cc6a75942328e8ac1594751326b072f3f"} Nov 02 08:49:36.331651 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:36.331651 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq" Nov 02 08:49:36.331878 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:36.331672 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c428a2386b8f306b1e04df5009a92c9cc6a75942328e8ac1594751326b072f3f" Nov 02 08:49:43.140838 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:49:43.140798 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:49:59.543803 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.543755 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h"] Nov 02 08:49:59.544337 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.544218 5899 cpu_manager.go:414] "containerMap: removing container" podUID="028899e5-16ed-4a11-b932-f5264ce2bae9" containerName="revoke-keys" Nov 02 08:49:59.544337 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.544233 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028899e5-16ed-4a11-b932-f5264ce2bae9" containerName="revoke-keys" Nov 02 08:49:59.544337 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.544328 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="028899e5-16ed-4a11-b932-f5264ce2bae9" containerName="revoke-keys" Nov 02 08:49:59.549044 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.549020 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" Nov 02 08:49:59.564731 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.564703 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:49:59.577810 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.577780 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:49:59.579861 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.579832 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h"] Nov 02 08:49:59.638386 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.638353 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-c667h\" (UID: \"d2d905e0-7907-473d-b7f6-f926adc0305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" Nov 02 08:49:59.638580 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.638419 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-c667h\" (UID: \"d2d905e0-7907-473d-b7f6-f926adc0305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" Nov 02 08:49:59.638580 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.638542 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47xv2\" (UniqueName: \"kubernetes.io/projected/d2d905e0-7907-473d-b7f6-f926adc0305c-kube-api-access-47xv2\") pod \"maas-api-revoke-sub-046e1343-c667h\" (UID: \"d2d905e0-7907-473d-b7f6-f926adc0305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" Nov 02 08:49:59.739961 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.739918 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47xv2\" (UniqueName: \"kubernetes.io/projected/d2d905e0-7907-473d-b7f6-f926adc0305c-kube-api-access-47xv2\") pod \"maas-api-revoke-sub-046e1343-c667h\" (UID: \"d2d905e0-7907-473d-b7f6-f926adc0305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" Nov 02 08:49:59.740203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.740023 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-c667h\" (UID: \"d2d905e0-7907-473d-b7f6-f926adc0305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" Nov 02 08:49:59.740203 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.740097 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-c667h\" (UID: \"d2d905e0-7907-473d-b7f6-f926adc0305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" Nov 02 08:49:59.740822 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.740796 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-c667h\" (UID: \"d2d905e0-7907-473d-b7f6-f926adc0305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" Nov 02 08:49:59.742371 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.742349 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-c667h\" (UID: \"d2d905e0-7907-473d-b7f6-f926adc0305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" Nov 02 08:49:59.764371 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.764339 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47xv2\" (UniqueName: \"kubernetes.io/projected/d2d905e0-7907-473d-b7f6-f926adc0305c-kube-api-access-47xv2\") pod \"maas-api-revoke-sub-046e1343-c667h\" (UID: \"d2d905e0-7907-473d-b7f6-f926adc0305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" Nov 02 08:49:59.862195 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:49:59.862095 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" Nov 02 08:50:00.191819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:00.191795 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h"] Nov 02 08:50:00.194315 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:50:00.194284 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d905e0_7907_473d_b7f6_f926adc0305c.slice/crio-666dafe7e98ff242e2a10e08e202b8b7fabd8716a68e814624392561ec297b0e WatchSource:0}: Error finding container 666dafe7e98ff242e2a10e08e202b8b7fabd8716a68e814624392561ec297b0e: Status 404 returned error can't find the container with id 666dafe7e98ff242e2a10e08e202b8b7fabd8716a68e814624392561ec297b0e Nov 02 08:50:00.492104 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:00.492063 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" event={"ID":"d2d905e0-7907-473d-b7f6-f926adc0305c","Type":"ContainerStarted","Data":"64ffbedb0d3356be7d0ad45e4fa1ab3ebd2ee5819b292a822cf0cf4f513bc81d"} Nov 02 08:50:00.492298 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:00.492125 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" event={"ID":"d2d905e0-7907-473d-b7f6-f926adc0305c","Type":"ContainerStarted","Data":"666dafe7e98ff242e2a10e08e202b8b7fabd8716a68e814624392561ec297b0e"} Nov 02 08:50:00.534448 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:00.534298 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" podStartSLOduration=1.5342821290000002 podStartE2EDuration="1.534282129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:50:00.533440463 +0000 UTC m=+5100.831721288" watchObservedRunningTime="2026-11-02 08:50:00.534282129 +0000 UTC m=+5100.832562957" Nov 02 08:50:01.498670 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:01.498633 5899 generic.go:356] "Generic (PLEG): container finished" podID="d2d905e0-7907-473d-b7f6-f926adc0305c" containerID="64ffbedb0d3356be7d0ad45e4fa1ab3ebd2ee5819b292a822cf0cf4f513bc81d" exitCode=0 Nov 02 08:50:01.499092 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:01.498724 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" event={"ID":"d2d905e0-7907-473d-b7f6-f926adc0305c","Type":"ContainerDied","Data":"64ffbedb0d3356be7d0ad45e4fa1ab3ebd2ee5819b292a822cf0cf4f513bc81d"} Nov 02 08:50:02.637098 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:02.637067 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" Nov 02 08:50:02.766932 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:02.766845 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-openshift-ca\") pod \"d2d905e0-7907-473d-b7f6-f926adc0305c\" (UID: \"d2d905e0-7907-473d-b7f6-f926adc0305c\") " Nov 02 08:50:02.766932 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:02.766933 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d905e0-7907-473d-b7f6-f926adc0305c-kube-api-access-47xv2\" (UniqueName: \"kubernetes.io/projected/d2d905e0-7907-473d-b7f6-f926adc0305c-kube-api-access-47xv2\") pod \"d2d905e0-7907-473d-b7f6-f926adc0305c\" (UID: \"d2d905e0-7907-473d-b7f6-f926adc0305c\") " Nov 02 08:50:02.767213 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:02.767016 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-xks-ca\") pod \"d2d905e0-7907-473d-b7f6-f926adc0305c\" (UID: \"d2d905e0-7907-473d-b7f6-f926adc0305c\") " Nov 02 08:50:02.767336 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:02.767310 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-openshift-ca" pod "d2d905e0-7907-473d-b7f6-f926adc0305c" (UID: "d2d905e0-7907-473d-b7f6-f926adc0305c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:50:02.769084 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:02.769058 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d905e0-7907-473d-b7f6-f926adc0305c-kube-api-access-47xv2" pod "d2d905e0-7907-473d-b7f6-f926adc0305c" (UID: "d2d905e0-7907-473d-b7f6-f926adc0305c"). InnerVolumeSpecName "kube-api-access-47xv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:02.769084 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:02.769071 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-xks-ca" pod "d2d905e0-7907-473d-b7f6-f926adc0305c" (UID: "d2d905e0-7907-473d-b7f6-f926adc0305c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:50:02.868800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:02.868763 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47xv2\" (UniqueName: \"kubernetes.io/projected/d2d905e0-7907-473d-b7f6-f926adc0305c-kube-api-access-47xv2\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:50:02.868800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:02.868794 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:50:02.868800 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:02.868805 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d905e0-7907-473d-b7f6-f926adc0305c-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:50:03.509624 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:03.509589 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" Nov 02 08:50:03.509783 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:03.509591 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h" event={"ID":"d2d905e0-7907-473d-b7f6-f926adc0305c","Type":"ContainerDied","Data":"666dafe7e98ff242e2a10e08e202b8b7fabd8716a68e814624392561ec297b0e"} Nov 02 08:50:03.509783 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:03.509719 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="666dafe7e98ff242e2a10e08e202b8b7fabd8716a68e814624392561ec297b0e" Nov 02 08:50:09.936186 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:09.936104 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5"] Nov 02 08:50:09.936669 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:09.936532 5899 cpu_manager.go:414] "containerMap: removing container" podUID="d2d905e0-7907-473d-b7f6-f926adc0305c" containerName="revoke-keys" Nov 02 08:50:09.936669 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:09.936545 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d905e0-7907-473d-b7f6-f926adc0305c" containerName="revoke-keys" Nov 02 08:50:09.936669 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:09.936640 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d905e0-7907-473d-b7f6-f926adc0305c" containerName="revoke-keys" Nov 02 08:50:09.939948 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:09.939926 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" Nov 02 08:50:09.970624 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:09.970580 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:50:09.970812 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:09.970661 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:50:09.972779 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:09.972754 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5"] Nov 02 08:50:10.032882 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:10.032847 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8pkw\" (UniqueName: \"kubernetes.io/projected/088d752b-fb3d-49cc-906e-164c6c1402ad-kube-api-access-w8pkw\") pod \"maas-api-revoke-sub-d3fba056-qvbs5\" (UID: \"088d752b-fb3d-49cc-906e-164c6c1402ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" Nov 02 08:50:10.033049 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:10.032892 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-qvbs5\" (UID: \"088d752b-fb3d-49cc-906e-164c6c1402ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" Nov 02 08:50:10.033049 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:10.032974 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-qvbs5\" (UID: \"088d752b-fb3d-49cc-906e-164c6c1402ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" Nov 02 08:50:10.134248 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:10.134208 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-qvbs5\" (UID: \"088d752b-fb3d-49cc-906e-164c6c1402ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" Nov 02 08:50:10.134455 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:10.134358 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8pkw\" (UniqueName: \"kubernetes.io/projected/088d752b-fb3d-49cc-906e-164c6c1402ad-kube-api-access-w8pkw\") pod \"maas-api-revoke-sub-d3fba056-qvbs5\" (UID: \"088d752b-fb3d-49cc-906e-164c6c1402ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" Nov 02 08:50:10.134455 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:10.134397 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-qvbs5\" (UID: \"088d752b-fb3d-49cc-906e-164c6c1402ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" Nov 02 08:50:10.135014 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:10.134989 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-qvbs5\" (UID: \"088d752b-fb3d-49cc-906e-164c6c1402ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" Nov 02 08:50:10.136408 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:10.136387 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-qvbs5\" (UID: \"088d752b-fb3d-49cc-906e-164c6c1402ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" Nov 02 08:50:10.158587 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:10.158562 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8pkw\" (UniqueName: \"kubernetes.io/projected/088d752b-fb3d-49cc-906e-164c6c1402ad-kube-api-access-w8pkw\") pod \"maas-api-revoke-sub-d3fba056-qvbs5\" (UID: \"088d752b-fb3d-49cc-906e-164c6c1402ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" Nov 02 08:50:10.255459 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:10.255419 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" Nov 02 08:50:10.554432 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:10.554388 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5"] Nov 02 08:50:10.557437 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:50:10.557406 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088d752b_fb3d_49cc_906e_164c6c1402ad.slice/crio-e40e123f7758b596b86312a34907723ec455e574f9c72ec8fa00a5a92c9d06a2 WatchSource:0}: Error finding container e40e123f7758b596b86312a34907723ec455e574f9c72ec8fa00a5a92c9d06a2: Status 404 returned error can't find the container with id e40e123f7758b596b86312a34907723ec455e574f9c72ec8fa00a5a92c9d06a2 Nov 02 08:50:11.556252 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:11.556209 5899 generic.go:356] "Generic (PLEG): container finished" podID="088d752b-fb3d-49cc-906e-164c6c1402ad" containerID="1134a057ef4af3451e57aaa87fe08ce55276caee2dcd6d9624ac276625a75f75" exitCode=0 Nov 02 08:50:11.556656 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:11.556303 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" event={"ID":"088d752b-fb3d-49cc-906e-164c6c1402ad","Type":"ContainerDied","Data":"1134a057ef4af3451e57aaa87fe08ce55276caee2dcd6d9624ac276625a75f75"} Nov 02 08:50:11.556656 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:11.556377 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" event={"ID":"088d752b-fb3d-49cc-906e-164c6c1402ad","Type":"ContainerStarted","Data":"e40e123f7758b596b86312a34907723ec455e574f9c72ec8fa00a5a92c9d06a2"} Nov 02 08:50:11.791815 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:11.791780 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:50:13.048070 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:13.048041 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" Nov 02 08:50:13.139760 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:13.139709 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-xks-ca\") pod \"088d752b-fb3d-49cc-906e-164c6c1402ad\" (UID: \"088d752b-fb3d-49cc-906e-164c6c1402ad\") " Nov 02 08:50:13.144404 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:13.140010 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-openshift-ca\") pod \"088d752b-fb3d-49cc-906e-164c6c1402ad\" (UID: \"088d752b-fb3d-49cc-906e-164c6c1402ad\") " Nov 02 08:50:13.144404 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:13.142595 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-openshift-ca" pod "088d752b-fb3d-49cc-906e-164c6c1402ad" (UID: "088d752b-fb3d-49cc-906e-164c6c1402ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:50:13.144404 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:13.143590 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088d752b-fb3d-49cc-906e-164c6c1402ad-kube-api-access-w8pkw\" (UniqueName: \"kubernetes.io/projected/088d752b-fb3d-49cc-906e-164c6c1402ad-kube-api-access-w8pkw\") pod \"088d752b-fb3d-49cc-906e-164c6c1402ad\" (UID: \"088d752b-fb3d-49cc-906e-164c6c1402ad\") " Nov 02 08:50:13.144404 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:13.143977 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:50:13.146026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:13.145976 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-xks-ca" pod "088d752b-fb3d-49cc-906e-164c6c1402ad" (UID: "088d752b-fb3d-49cc-906e-164c6c1402ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:50:13.146438 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:13.146401 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088d752b-fb3d-49cc-906e-164c6c1402ad-kube-api-access-w8pkw" pod "088d752b-fb3d-49cc-906e-164c6c1402ad" (UID: "088d752b-fb3d-49cc-906e-164c6c1402ad"). InnerVolumeSpecName "kube-api-access-w8pkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:13.237734 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:50:13.237628 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:50:13.244741 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:13.244705 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8pkw\" (UniqueName: \"kubernetes.io/projected/088d752b-fb3d-49cc-906e-164c6c1402ad-kube-api-access-w8pkw\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:50:13.244741 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:13.244736 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088d752b-fb3d-49cc-906e-164c6c1402ad-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:50:13.628847 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:13.628694 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" event={"ID":"088d752b-fb3d-49cc-906e-164c6c1402ad","Type":"ContainerDied","Data":"e40e123f7758b596b86312a34907723ec455e574f9c72ec8fa00a5a92c9d06a2"} Nov 02 08:50:13.628847 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:13.628804 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e40e123f7758b596b86312a34907723ec455e574f9c72ec8fa00a5a92c9d06a2" Nov 02 08:50:13.629250 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:13.629232 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5" Nov 02 08:50:31.545007 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.544970 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j"] Nov 02 08:50:31.545479 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.545410 5899 cpu_manager.go:414] "containerMap: removing container" podUID="088d752b-fb3d-49cc-906e-164c6c1402ad" containerName="revoke-keys" Nov 02 08:50:31.545479 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.545424 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088d752b-fb3d-49cc-906e-164c6c1402ad" containerName="revoke-keys" Nov 02 08:50:31.545570 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.545519 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="088d752b-fb3d-49cc-906e-164c6c1402ad" containerName="revoke-keys" Nov 02 08:50:31.551991 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.551969 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" Nov 02 08:50:31.575926 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.575864 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j"] Nov 02 08:50:31.585818 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.585777 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:50:31.586027 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.585792 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:50:31.668017 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.667974 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-pbw6j\" (UID: \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" Nov 02 08:50:31.668251 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.668137 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws9h6\" (UniqueName: \"kubernetes.io/projected/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-kube-api-access-ws9h6\") pod \"maas-api-revoke-sub-ae594979-pbw6j\" (UID: \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" Nov 02 08:50:31.668306 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.668264 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-pbw6j\" (UID: \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" Nov 02 08:50:31.769777 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.769734 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws9h6\" (UniqueName: \"kubernetes.io/projected/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-kube-api-access-ws9h6\") pod \"maas-api-revoke-sub-ae594979-pbw6j\" (UID: \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" Nov 02 08:50:31.769962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.769801 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-pbw6j\" (UID: \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" Nov 02 08:50:31.769962 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.769827 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-pbw6j\" (UID: \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" Nov 02 08:50:31.770565 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.770542 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-pbw6j\" (UID: \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" Nov 02 08:50:31.772130 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.772112 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-pbw6j\" (UID: \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" Nov 02 08:50:31.798308 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.798236 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws9h6\" (UniqueName: \"kubernetes.io/projected/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-kube-api-access-ws9h6\") pod \"maas-api-revoke-sub-ae594979-pbw6j\" (UID: \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" Nov 02 08:50:31.865333 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:31.865297 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" Nov 02 08:50:32.168580 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:32.168552 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j"] Nov 02 08:50:32.170821 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:50:32.170791 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ca6a5c_d247_43aa_afb1_15a0bcb985f1.slice/crio-9705feb7c085434b2fbab9304ac84a19fbfeb38add1b893fd1601245e54f7883 WatchSource:0}: Error finding container 9705feb7c085434b2fbab9304ac84a19fbfeb38add1b893fd1601245e54f7883: Status 404 returned error can't find the container with id 9705feb7c085434b2fbab9304ac84a19fbfeb38add1b893fd1601245e54f7883 Nov 02 08:50:32.759453 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:32.759408 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" event={"ID":"76ca6a5c-d247-43aa-afb1-15a0bcb985f1","Type":"ContainerStarted","Data":"6f3d0cf227eaca2c7b6c1de3f380678c58f09d2c48f005eb69cbac77bbff5c6a"} Nov 02 08:50:32.759859 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:32.759478 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" event={"ID":"76ca6a5c-d247-43aa-afb1-15a0bcb985f1","Type":"ContainerStarted","Data":"9705feb7c085434b2fbab9304ac84a19fbfeb38add1b893fd1601245e54f7883"} Nov 02 08:50:32.804058 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:32.803997 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" podStartSLOduration=1.803980595 podStartE2EDuration="1.803980595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:50:32.803663179 +0000 UTC m=+5133.101944007" watchObservedRunningTime="2026-11-02 08:50:32.803980595 +0000 UTC m=+5133.102261424" Nov 02 08:50:33.766535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:33.766499 5899 generic.go:356] "Generic (PLEG): container finished" podID="76ca6a5c-d247-43aa-afb1-15a0bcb985f1" containerID="6f3d0cf227eaca2c7b6c1de3f380678c58f09d2c48f005eb69cbac77bbff5c6a" exitCode=0 Nov 02 08:50:33.766935 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:33.766584 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" event={"ID":"76ca6a5c-d247-43aa-afb1-15a0bcb985f1","Type":"ContainerDied","Data":"6f3d0cf227eaca2c7b6c1de3f380678c58f09d2c48f005eb69cbac77bbff5c6a"} Nov 02 08:50:34.907433 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:34.907406 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" Nov 02 08:50:35.001098 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:35.001058 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-openshift-ca\") pod \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\" (UID: \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\") " Nov 02 08:50:35.001331 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:35.001124 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-kube-api-access-ws9h6\" (UniqueName: \"kubernetes.io/projected/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-kube-api-access-ws9h6\") pod \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\" (UID: \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\") " Nov 02 08:50:35.001331 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:35.001246 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-xks-ca\") pod \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\" (UID: \"76ca6a5c-d247-43aa-afb1-15a0bcb985f1\") " Nov 02 08:50:35.001497 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:35.001472 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-openshift-ca" pod "76ca6a5c-d247-43aa-afb1-15a0bcb985f1" (UID: "76ca6a5c-d247-43aa-afb1-15a0bcb985f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:50:35.003292 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:35.003265 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-xks-ca" pod "76ca6a5c-d247-43aa-afb1-15a0bcb985f1" (UID: "76ca6a5c-d247-43aa-afb1-15a0bcb985f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:50:35.003372 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:35.003323 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-kube-api-access-ws9h6" pod "76ca6a5c-d247-43aa-afb1-15a0bcb985f1" (UID: "76ca6a5c-d247-43aa-afb1-15a0bcb985f1"). InnerVolumeSpecName "kube-api-access-ws9h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:50:35.102568 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:35.102491 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:50:35.102568 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:35.102516 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:50:35.102568 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:35.102526 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws9h6\" (UniqueName: \"kubernetes.io/projected/76ca6a5c-d247-43aa-afb1-15a0bcb985f1-kube-api-access-ws9h6\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:50:35.778280 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:35.778245 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" Nov 02 08:50:35.778448 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:35.778281 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j" event={"ID":"76ca6a5c-d247-43aa-afb1-15a0bcb985f1","Type":"ContainerDied","Data":"9705feb7c085434b2fbab9304ac84a19fbfeb38add1b893fd1601245e54f7883"} Nov 02 08:50:35.778448 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:35.778338 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9705feb7c085434b2fbab9304ac84a19fbfeb38add1b893fd1601245e54f7883" Nov 02 08:50:37.028491 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:37.028446 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj"] Nov 02 08:50:37.041495 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:37.041468 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p2slj"] Nov 02 08:50:37.819106 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:37.819070 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c37fa8-67f8-4884-9b11-2a75bdf6f137" path="/var/lib/kubelet/pods/a7c37fa8-67f8-4884-9b11-2a75bdf6f137/volumes" Nov 02 08:50:43.141413 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:50:43.141326 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:50:54.028844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:54.028804 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj"] Nov 02 08:50:54.041769 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:54.041736 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p2vlj"] Nov 02 08:50:55.818087 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:55.818055 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7" path="/var/lib/kubelet/pods/c54a97ae-1bf3-43b5-9d6b-3a561b0f9ed7/volumes" Nov 02 08:50:58.557596 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.557558 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg"] Nov 02 08:50:58.558013 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.558000 5899 cpu_manager.go:414] "containerMap: removing container" podUID="76ca6a5c-d247-43aa-afb1-15a0bcb985f1" containerName="revoke-keys" Nov 02 08:50:58.558059 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.558014 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ca6a5c-d247-43aa-afb1-15a0bcb985f1" containerName="revoke-keys" Nov 02 08:50:58.558131 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.558121 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ca6a5c-d247-43aa-afb1-15a0bcb985f1" containerName="revoke-keys" Nov 02 08:50:58.561514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.561496 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" Nov 02 08:50:58.580476 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.580445 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg"] Nov 02 08:50:58.591147 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.591119 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:50:58.591295 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.591208 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:50:58.649244 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.649195 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4dvdg\" (UID: \"a9bb891f-fa7a-4722-93ee-6785cd875939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" Nov 02 08:50:58.649431 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.649379 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4dvdg\" (UID: \"a9bb891f-fa7a-4722-93ee-6785cd875939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" Nov 02 08:50:58.649431 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.649415 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtfgm\" (UniqueName: \"kubernetes.io/projected/a9bb891f-fa7a-4722-93ee-6785cd875939-kube-api-access-jtfgm\") pod \"maas-api-revoke-sub-a4aef7d7-4dvdg\" (UID: \"a9bb891f-fa7a-4722-93ee-6785cd875939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" Nov 02 08:50:58.750939 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.750899 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4dvdg\" (UID: \"a9bb891f-fa7a-4722-93ee-6785cd875939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" Nov 02 08:50:58.751103 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.750951 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtfgm\" (UniqueName: \"kubernetes.io/projected/a9bb891f-fa7a-4722-93ee-6785cd875939-kube-api-access-jtfgm\") pod \"maas-api-revoke-sub-a4aef7d7-4dvdg\" (UID: \"a9bb891f-fa7a-4722-93ee-6785cd875939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" Nov 02 08:50:58.751103 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.750995 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4dvdg\" (UID: \"a9bb891f-fa7a-4722-93ee-6785cd875939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" Nov 02 08:50:58.751763 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.751738 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4dvdg\" (UID: \"a9bb891f-fa7a-4722-93ee-6785cd875939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" Nov 02 08:50:58.753356 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.753332 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4dvdg\" (UID: \"a9bb891f-fa7a-4722-93ee-6785cd875939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" Nov 02 08:50:58.775629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.775600 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtfgm\" (UniqueName: \"kubernetes.io/projected/a9bb891f-fa7a-4722-93ee-6785cd875939-kube-api-access-jtfgm\") pod \"maas-api-revoke-sub-a4aef7d7-4dvdg\" (UID: \"a9bb891f-fa7a-4722-93ee-6785cd875939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" Nov 02 08:50:58.874892 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:58.874811 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" Nov 02 08:50:59.174261 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:59.174201 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg"] Nov 02 08:50:59.176967 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:50:59.176942 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bb891f_fa7a_4722_93ee_6785cd875939.slice/crio-4e66250d13b1179063b093700a7955eab1ab43befce806fc0ede9059beec2ac5 WatchSource:0}: Error finding container 4e66250d13b1179063b093700a7955eab1ab43befce806fc0ede9059beec2ac5: Status 404 returned error can't find the container with id 4e66250d13b1179063b093700a7955eab1ab43befce806fc0ede9059beec2ac5 Nov 02 08:50:59.910106 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:59.910064 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" event={"ID":"a9bb891f-fa7a-4722-93ee-6785cd875939","Type":"ContainerStarted","Data":"fe068a9c2f23bcb9cbc2a2c8a8fdbe134c83dcb5c208bac2b27b39fb0947b284"} Nov 02 08:50:59.910591 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:59.910129 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" event={"ID":"a9bb891f-fa7a-4722-93ee-6785cd875939","Type":"ContainerStarted","Data":"4e66250d13b1179063b093700a7955eab1ab43befce806fc0ede9059beec2ac5"} Nov 02 08:50:59.953539 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:50:59.953486 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" podStartSLOduration=1.9534645259999999 podStartE2EDuration="1.953464526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:50:59.952947539 +0000 UTC m=+5160.251228364" watchObservedRunningTime="2026-11-02 08:50:59.953464526 +0000 UTC m=+5160.251745350" Nov 02 08:51:00.916533 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:00.916495 5899 generic.go:356] "Generic (PLEG): container finished" podID="a9bb891f-fa7a-4722-93ee-6785cd875939" containerID="fe068a9c2f23bcb9cbc2a2c8a8fdbe134c83dcb5c208bac2b27b39fb0947b284" exitCode=0 Nov 02 08:51:00.916937 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:00.916592 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" event={"ID":"a9bb891f-fa7a-4722-93ee-6785cd875939","Type":"ContainerDied","Data":"fe068a9c2f23bcb9cbc2a2c8a8fdbe134c83dcb5c208bac2b27b39fb0947b284"} Nov 02 08:51:01.513691 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:01.513658 5899 scope.go:122] "RemoveContainer" containerID="dddfccfaeba6e25d48c3c8e4f8de421c52740323a9932df2df57d72936db283f" Nov 02 08:51:02.080979 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:02.080950 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" Nov 02 08:51:02.184404 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:02.184322 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-xks-ca\") pod \"a9bb891f-fa7a-4722-93ee-6785cd875939\" (UID: \"a9bb891f-fa7a-4722-93ee-6785cd875939\") " Nov 02 08:51:02.184578 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:02.184407 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bb891f-fa7a-4722-93ee-6785cd875939-kube-api-access-jtfgm\" (UniqueName: \"kubernetes.io/projected/a9bb891f-fa7a-4722-93ee-6785cd875939-kube-api-access-jtfgm\") pod \"a9bb891f-fa7a-4722-93ee-6785cd875939\" (UID: \"a9bb891f-fa7a-4722-93ee-6785cd875939\") " Nov 02 08:51:02.184578 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:02.184439 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-openshift-ca\") pod \"a9bb891f-fa7a-4722-93ee-6785cd875939\" (UID: \"a9bb891f-fa7a-4722-93ee-6785cd875939\") " Nov 02 08:51:02.184841 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:02.184820 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-openshift-ca" pod "a9bb891f-fa7a-4722-93ee-6785cd875939" (UID: "a9bb891f-fa7a-4722-93ee-6785cd875939"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:51:02.186514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:02.186494 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bb891f-fa7a-4722-93ee-6785cd875939-kube-api-access-jtfgm" pod "a9bb891f-fa7a-4722-93ee-6785cd875939" (UID: "a9bb891f-fa7a-4722-93ee-6785cd875939"). InnerVolumeSpecName "kube-api-access-jtfgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:51:02.186588 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:02.186526 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-xks-ca" pod "a9bb891f-fa7a-4722-93ee-6785cd875939" (UID: "a9bb891f-fa7a-4722-93ee-6785cd875939"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:51:02.286052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:02.286013 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:51:02.286052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:02.286047 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtfgm\" (UniqueName: \"kubernetes.io/projected/a9bb891f-fa7a-4722-93ee-6785cd875939-kube-api-access-jtfgm\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:51:02.286052 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:02.286058 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9bb891f-fa7a-4722-93ee-6785cd875939-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:51:02.928684 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:02.928649 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" event={"ID":"a9bb891f-fa7a-4722-93ee-6785cd875939","Type":"ContainerDied","Data":"4e66250d13b1179063b093700a7955eab1ab43befce806fc0ede9059beec2ac5"} Nov 02 08:51:02.928684 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:02.928679 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg" Nov 02 08:51:02.928907 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:02.928700 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e66250d13b1179063b093700a7955eab1ab43befce806fc0ede9059beec2ac5" Nov 02 08:51:07.650566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:07.650530 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:11.924429 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:11.924396 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:51:13.140729 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:51:13.140687 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:51:14.032501 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:14.032463 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw"] Nov 02 08:51:14.046793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:14.046759 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc45422-gqxbw"] Nov 02 08:51:15.501540 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.501493 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g"] Nov 02 08:51:15.502112 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.502092 5899 cpu_manager.go:414] "containerMap: removing container" podUID="a9bb891f-fa7a-4722-93ee-6785cd875939" containerName="revoke-keys" Nov 02 08:51:15.502226 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.502114 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9bb891f-fa7a-4722-93ee-6785cd875939" containerName="revoke-keys" Nov 02 08:51:15.502319 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.502305 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9bb891f-fa7a-4722-93ee-6785cd875939" containerName="revoke-keys" Nov 02 08:51:15.505510 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.505488 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" Nov 02 08:51:15.524696 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.524670 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:51:15.538113 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.538055 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:51:15.541365 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.541327 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g"] Nov 02 08:51:15.620318 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.620277 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqdzr\" (UniqueName: \"kubernetes.io/projected/c8126dd7-bce4-4574-afac-4b952c95a9c2-kube-api-access-cqdzr\") pod \"maas-api-revoke-sub-79805ca9-qbr7g\" (UID: \"c8126dd7-bce4-4574-afac-4b952c95a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" Nov 02 08:51:15.620548 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.620456 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qbr7g\" (UID: \"c8126dd7-bce4-4574-afac-4b952c95a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" Nov 02 08:51:15.620548 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.620506 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qbr7g\" (UID: \"c8126dd7-bce4-4574-afac-4b952c95a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" Nov 02 08:51:15.721375 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.721324 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qbr7g\" (UID: \"c8126dd7-bce4-4574-afac-4b952c95a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" Nov 02 08:51:15.721375 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.721382 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qbr7g\" (UID: \"c8126dd7-bce4-4574-afac-4b952c95a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" Nov 02 08:51:15.721659 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.721473 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdzr\" (UniqueName: \"kubernetes.io/projected/c8126dd7-bce4-4574-afac-4b952c95a9c2-kube-api-access-cqdzr\") pod \"maas-api-revoke-sub-79805ca9-qbr7g\" (UID: \"c8126dd7-bce4-4574-afac-4b952c95a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" Nov 02 08:51:15.722302 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.722270 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qbr7g\" (UID: \"c8126dd7-bce4-4574-afac-4b952c95a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" Nov 02 08:51:15.723963 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.723934 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qbr7g\" (UID: \"c8126dd7-bce4-4574-afac-4b952c95a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" Nov 02 08:51:15.745988 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.745955 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqdzr\" (UniqueName: \"kubernetes.io/projected/c8126dd7-bce4-4574-afac-4b952c95a9c2-kube-api-access-cqdzr\") pod \"maas-api-revoke-sub-79805ca9-qbr7g\" (UID: \"c8126dd7-bce4-4574-afac-4b952c95a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" Nov 02 08:51:15.818674 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.818582 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd6f7114-c90d-4fc9-96a7-c955fd1c87aa" path="/var/lib/kubelet/pods/cd6f7114-c90d-4fc9-96a7-c955fd1c87aa/volumes" Nov 02 08:51:15.821657 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:15.821625 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" Nov 02 08:51:16.227314 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:16.226391 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g"] Nov 02 08:51:16.240468 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:51:16.240432 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8126dd7_bce4_4574_afac_4b952c95a9c2.slice/crio-ac6b6134a68dc87c9b1b126334fa22b7ec07f9792fe7ab07d85834139606eade WatchSource:0}: Error finding container ac6b6134a68dc87c9b1b126334fa22b7ec07f9792fe7ab07d85834139606eade: Status 404 returned error can't find the container with id ac6b6134a68dc87c9b1b126334fa22b7ec07f9792fe7ab07d85834139606eade Nov 02 08:51:17.053460 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:17.053408 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" event={"ID":"c8126dd7-bce4-4574-afac-4b952c95a9c2","Type":"ContainerStarted","Data":"37855e38c3d1126333e24c9c1a893bbd53d2dc289143722c9bc8addc23eefd42"} Nov 02 08:51:17.053890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:17.053482 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" event={"ID":"c8126dd7-bce4-4574-afac-4b952c95a9c2","Type":"ContainerStarted","Data":"ac6b6134a68dc87c9b1b126334fa22b7ec07f9792fe7ab07d85834139606eade"} Nov 02 08:51:17.112186 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:17.104130 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" podStartSLOduration=2.104110161 podStartE2EDuration="2.104110161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:51:17.10290344 +0000 UTC m=+5177.401184263" watchObservedRunningTime="2026-11-02 08:51:17.104110161 +0000 UTC m=+5177.402390989" Nov 02 08:51:18.060633 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:18.060595 5899 generic.go:356] "Generic (PLEG): container finished" podID="c8126dd7-bce4-4574-afac-4b952c95a9c2" containerID="37855e38c3d1126333e24c9c1a893bbd53d2dc289143722c9bc8addc23eefd42" exitCode=0 Nov 02 08:51:18.061071 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:18.060687 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" event={"ID":"c8126dd7-bce4-4574-afac-4b952c95a9c2","Type":"ContainerDied","Data":"37855e38c3d1126333e24c9c1a893bbd53d2dc289143722c9bc8addc23eefd42"} Nov 02 08:51:19.203138 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:19.203110 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" Nov 02 08:51:19.267937 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:19.267902 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8126dd7-bce4-4574-afac-4b952c95a9c2-kube-api-access-cqdzr\" (UniqueName: \"kubernetes.io/projected/c8126dd7-bce4-4574-afac-4b952c95a9c2-kube-api-access-cqdzr\") pod \"c8126dd7-bce4-4574-afac-4b952c95a9c2\" (UID: \"c8126dd7-bce4-4574-afac-4b952c95a9c2\") " Nov 02 08:51:19.268145 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:19.268005 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-openshift-ca\") pod \"c8126dd7-bce4-4574-afac-4b952c95a9c2\" (UID: \"c8126dd7-bce4-4574-afac-4b952c95a9c2\") " Nov 02 08:51:19.268145 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:19.268026 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-xks-ca\") pod \"c8126dd7-bce4-4574-afac-4b952c95a9c2\" (UID: \"c8126dd7-bce4-4574-afac-4b952c95a9c2\") " Nov 02 08:51:19.268470 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:19.268442 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-openshift-ca" pod "c8126dd7-bce4-4574-afac-4b952c95a9c2" (UID: "c8126dd7-bce4-4574-afac-4b952c95a9c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:51:19.270312 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:19.270280 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-xks-ca" pod "c8126dd7-bce4-4574-afac-4b952c95a9c2" (UID: "c8126dd7-bce4-4574-afac-4b952c95a9c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:51:19.270451 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:19.270426 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8126dd7-bce4-4574-afac-4b952c95a9c2-kube-api-access-cqdzr" pod "c8126dd7-bce4-4574-afac-4b952c95a9c2" (UID: "c8126dd7-bce4-4574-afac-4b952c95a9c2"). InnerVolumeSpecName "kube-api-access-cqdzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:51:19.369217 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:19.369085 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:51:19.369217 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:19.369121 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8126dd7-bce4-4574-afac-4b952c95a9c2-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:51:19.369217 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:19.369134 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqdzr\" (UniqueName: \"kubernetes.io/projected/c8126dd7-bce4-4574-afac-4b952c95a9c2-kube-api-access-cqdzr\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:51:20.072233 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:20.072197 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" Nov 02 08:51:20.072233 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:20.072219 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g" event={"ID":"c8126dd7-bce4-4574-afac-4b952c95a9c2","Type":"ContainerDied","Data":"ac6b6134a68dc87c9b1b126334fa22b7ec07f9792fe7ab07d85834139606eade"} Nov 02 08:51:20.072440 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:20.072265 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac6b6134a68dc87c9b1b126334fa22b7ec07f9792fe7ab07d85834139606eade" Nov 02 08:51:29.223281 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.223223 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4"] Nov 02 08:51:29.224622 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.224593 5899 cpu_manager.go:414] "containerMap: removing container" podUID="c8126dd7-bce4-4574-afac-4b952c95a9c2" containerName="revoke-keys" Nov 02 08:51:29.224789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.224773 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8126dd7-bce4-4574-afac-4b952c95a9c2" containerName="revoke-keys" Nov 02 08:51:29.225047 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.225031 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8126dd7-bce4-4574-afac-4b952c95a9c2" containerName="revoke-keys" Nov 02 08:51:29.233153 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.230814 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" Nov 02 08:51:29.243377 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.243349 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4"] Nov 02 08:51:29.253973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.253946 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:51:29.253973 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.253948 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:51:29.390882 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.390826 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jrx2\" (UniqueName: \"kubernetes.io/projected/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-kube-api-access-2jrx2\") pod \"maas-api-revoke-sub-0b7e8edb-clhf4\" (UID: \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" Nov 02 08:51:29.391226 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.391199 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-clhf4\" (UID: \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" Nov 02 08:51:29.391566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.391544 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-clhf4\" (UID: \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" Nov 02 08:51:29.493945 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.493880 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jrx2\" (UniqueName: \"kubernetes.io/projected/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-kube-api-access-2jrx2\") pod \"maas-api-revoke-sub-0b7e8edb-clhf4\" (UID: \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" Nov 02 08:51:29.494772 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.494726 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-clhf4\" (UID: \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" Nov 02 08:51:29.518668 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.495762 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-clhf4\" (UID: \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" Nov 02 08:51:29.519175 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.519114 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-clhf4\" (UID: \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" Nov 02 08:51:29.525095 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.523921 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-clhf4\" (UID: \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" Nov 02 08:51:29.533405 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.533346 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jrx2\" (UniqueName: \"kubernetes.io/projected/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-kube-api-access-2jrx2\") pod \"maas-api-revoke-sub-0b7e8edb-clhf4\" (UID: \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" Nov 02 08:51:29.548953 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.548920 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" Nov 02 08:51:29.939087 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:29.939004 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4"] Nov 02 08:51:29.943016 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:51:29.942990 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49ddd58_9d92_4fa3_b592_ca66ae1191d2.slice/crio-e87f24209e0bf9d7cace751ca55bc436a1f75c5c6502777eff7f66567f3083ca WatchSource:0}: Error finding container e87f24209e0bf9d7cace751ca55bc436a1f75c5c6502777eff7f66567f3083ca: Status 404 returned error can't find the container with id e87f24209e0bf9d7cace751ca55bc436a1f75c5c6502777eff7f66567f3083ca Nov 02 08:51:30.137080 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:30.137033 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" event={"ID":"b49ddd58-9d92-4fa3-b592-ca66ae1191d2","Type":"ContainerStarted","Data":"e87f24209e0bf9d7cace751ca55bc436a1f75c5c6502777eff7f66567f3083ca"} Nov 02 08:51:31.144221 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:31.144154 5899 generic.go:356] "Generic (PLEG): container finished" podID="b49ddd58-9d92-4fa3-b592-ca66ae1191d2" containerID="3d5e1f925a33a99f54368c1a222b85fee95cb67c20740396715c03e886d930ad" exitCode=0 Nov 02 08:51:31.144630 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:31.144249 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" event={"ID":"b49ddd58-9d92-4fa3-b592-ca66ae1191d2","Type":"ContainerDied","Data":"3d5e1f925a33a99f54368c1a222b85fee95cb67c20740396715c03e886d930ad"} Nov 02 08:51:32.288859 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:32.288828 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" Nov 02 08:51:32.363812 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:32.363775 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-openshift-ca\") pod \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\" (UID: \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\") " Nov 02 08:51:32.364012 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:32.363848 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-kube-api-access-2jrx2\" (UniqueName: \"kubernetes.io/projected/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-kube-api-access-2jrx2\") pod \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\" (UID: \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\") " Nov 02 08:51:32.364012 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:32.363874 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-xks-ca\") pod \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\" (UID: \"b49ddd58-9d92-4fa3-b592-ca66ae1191d2\") " Nov 02 08:51:32.364260 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:32.364231 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-openshift-ca" pod "b49ddd58-9d92-4fa3-b592-ca66ae1191d2" (UID: "b49ddd58-9d92-4fa3-b592-ca66ae1191d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:51:32.366008 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:32.365980 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-xks-ca" pod "b49ddd58-9d92-4fa3-b592-ca66ae1191d2" (UID: "b49ddd58-9d92-4fa3-b592-ca66ae1191d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:51:32.366091 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:32.366072 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-kube-api-access-2jrx2" pod "b49ddd58-9d92-4fa3-b592-ca66ae1191d2" (UID: "b49ddd58-9d92-4fa3-b592-ca66ae1191d2"). InnerVolumeSpecName "kube-api-access-2jrx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:51:32.465542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:32.465454 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jrx2\" (UniqueName: \"kubernetes.io/projected/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-kube-api-access-2jrx2\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:51:32.465542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:32.465490 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:51:32.465542 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:32.465503 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49ddd58-9d92-4fa3-b592-ca66ae1191d2-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:51:33.156924 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:33.156889 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" Nov 02 08:51:33.157116 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:33.156896 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4" event={"ID":"b49ddd58-9d92-4fa3-b592-ca66ae1191d2","Type":"ContainerDied","Data":"e87f24209e0bf9d7cace751ca55bc436a1f75c5c6502777eff7f66567f3083ca"} Nov 02 08:51:33.157116 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:33.157016 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e87f24209e0bf9d7cace751ca55bc436a1f75c5c6502777eff7f66567f3083ca" Nov 02 08:51:40.012616 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.012578 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh"] Nov 02 08:51:40.013036 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.013022 5899 cpu_manager.go:414] "containerMap: removing container" podUID="b49ddd58-9d92-4fa3-b592-ca66ae1191d2" containerName="revoke-keys" Nov 02 08:51:40.013036 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.013034 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49ddd58-9d92-4fa3-b592-ca66ae1191d2" containerName="revoke-keys" Nov 02 08:51:40.013121 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.013111 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49ddd58-9d92-4fa3-b592-ca66ae1191d2" containerName="revoke-keys" Nov 02 08:51:40.016610 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.016588 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" Nov 02 08:51:40.035397 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.035370 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:51:40.047570 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.047530 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:51:40.049777 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.049749 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh"] Nov 02 08:51:40.142205 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.142145 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4p7zh\" (UID: \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" Nov 02 08:51:40.142398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.142215 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twcvv\" (UniqueName: \"kubernetes.io/projected/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-kube-api-access-twcvv\") pod \"maas-api-revoke-sub-3d4156f1-4p7zh\" (UID: \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" Nov 02 08:51:40.142398 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.142320 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4p7zh\" (UID: \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" Nov 02 08:51:40.243394 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.243351 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4p7zh\" (UID: \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" Nov 02 08:51:40.243394 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.243401 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twcvv\" (UniqueName: \"kubernetes.io/projected/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-kube-api-access-twcvv\") pod \"maas-api-revoke-sub-3d4156f1-4p7zh\" (UID: \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" Nov 02 08:51:40.243657 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.243452 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4p7zh\" (UID: \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" Nov 02 08:51:40.244083 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.244059 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4p7zh\" (UID: \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" Nov 02 08:51:40.245718 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.245696 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4p7zh\" (UID: \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" Nov 02 08:51:40.267966 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.267902 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twcvv\" (UniqueName: \"kubernetes.io/projected/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-kube-api-access-twcvv\") pod \"maas-api-revoke-sub-3d4156f1-4p7zh\" (UID: \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" Nov 02 08:51:40.329514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.329475 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" Nov 02 08:51:40.629692 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:40.629480 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh"] Nov 02 08:51:40.632210 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:51:40.632174 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7a6b61_34cf_4bd4_bfe3_88890b6a42f6.slice/crio-d6ca0b03ad1a60cd6e91168bdc2e0e14dca9f35640cc3fa1800b607788d73c48 WatchSource:0}: Error finding container d6ca0b03ad1a60cd6e91168bdc2e0e14dca9f35640cc3fa1800b607788d73c48: Status 404 returned error can't find the container with id d6ca0b03ad1a60cd6e91168bdc2e0e14dca9f35640cc3fa1800b607788d73c48 Nov 02 08:51:41.198143 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:41.198054 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" event={"ID":"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6","Type":"ContainerStarted","Data":"17faebfd239fb470412b639426b6e528be7620b577406fd453038b7120af9cc6"} Nov 02 08:51:41.198143 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:41.198113 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" event={"ID":"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6","Type":"ContainerStarted","Data":"d6ca0b03ad1a60cd6e91168bdc2e0e14dca9f35640cc3fa1800b607788d73c48"} Nov 02 08:51:41.242278 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:41.242220 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" podStartSLOduration=2.242205335 podStartE2EDuration="2.242205335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:51:41.240812634 +0000 UTC m=+5201.539093459" watchObservedRunningTime="2026-11-02 08:51:41.242205335 +0000 UTC m=+5201.540486159" Nov 02 08:51:42.205230 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:42.205194 5899 generic.go:356] "Generic (PLEG): container finished" podID="3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6" containerID="17faebfd239fb470412b639426b6e528be7620b577406fd453038b7120af9cc6" exitCode=0 Nov 02 08:51:42.205718 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:42.205331 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" event={"ID":"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6","Type":"ContainerDied","Data":"17faebfd239fb470412b639426b6e528be7620b577406fd453038b7120af9cc6"} Nov 02 08:51:43.140944 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:51:43.140908 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:51:43.346298 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:43.346267 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" Nov 02 08:51:43.482456 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:43.482424 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-kube-api-access-twcvv\" (UniqueName: \"kubernetes.io/projected/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-kube-api-access-twcvv\") pod \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\" (UID: \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\") " Nov 02 08:51:43.482664 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:43.482501 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-xks-ca\") pod \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\" (UID: \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\") " Nov 02 08:51:43.482664 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:43.482576 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-openshift-ca\") pod \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\" (UID: \"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6\") " Nov 02 08:51:43.482989 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:43.482964 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-openshift-ca" pod "3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6" (UID: "3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:51:43.484685 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:43.484661 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-xks-ca" pod "3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6" (UID: "3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:51:43.484780 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:43.484757 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-kube-api-access-twcvv" pod "3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6" (UID: "3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6"). InnerVolumeSpecName "kube-api-access-twcvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:51:43.583874 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:43.583840 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:51:43.583874 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:43.583869 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twcvv\" (UniqueName: \"kubernetes.io/projected/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-kube-api-access-twcvv\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:51:43.583874 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:43.583880 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:51:44.216606 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:44.216519 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" Nov 02 08:51:44.216606 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:44.216559 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh" event={"ID":"3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6","Type":"ContainerDied","Data":"d6ca0b03ad1a60cd6e91168bdc2e0e14dca9f35640cc3fa1800b607788d73c48"} Nov 02 08:51:44.216848 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:44.216610 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6ca0b03ad1a60cd6e91168bdc2e0e14dca9f35640cc3fa1800b607788d73c48" Nov 02 08:51:58.274523 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.274468 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh"] Nov 02 08:51:58.286791 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.286760 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4p7zh"] Nov 02 08:51:58.316354 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.316321 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2"] Nov 02 08:51:58.316774 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.316760 5899 cpu_manager.go:414] "containerMap: removing container" podUID="3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6" containerName="revoke-keys" Nov 02 08:51:58.316822 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.316778 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6" containerName="revoke-keys" Nov 02 08:51:58.316895 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.316884 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6" containerName="revoke-keys" Nov 02 08:51:58.320347 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.320328 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" Nov 02 08:51:58.340613 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.340561 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:51:58.340781 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.340626 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:51:58.342871 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.342841 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2"] Nov 02 08:51:58.398636 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.398597 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sgcn\" (UniqueName: \"kubernetes.io/projected/d27d1869-2b4d-45c9-b878-4e9b912358c6-kube-api-access-5sgcn\") pod \"maas-api-revoke-sub-3d4156f1-f2cx2\" (UID: \"d27d1869-2b4d-45c9-b878-4e9b912358c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" Nov 02 08:51:58.398821 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.398670 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f2cx2\" (UID: \"d27d1869-2b4d-45c9-b878-4e9b912358c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" Nov 02 08:51:58.398887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.398809 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f2cx2\" (UID: \"d27d1869-2b4d-45c9-b878-4e9b912358c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" Nov 02 08:51:58.500013 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.499967 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f2cx2\" (UID: \"d27d1869-2b4d-45c9-b878-4e9b912358c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" Nov 02 08:51:58.500208 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.500080 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sgcn\" (UniqueName: \"kubernetes.io/projected/d27d1869-2b4d-45c9-b878-4e9b912358c6-kube-api-access-5sgcn\") pod \"maas-api-revoke-sub-3d4156f1-f2cx2\" (UID: \"d27d1869-2b4d-45c9-b878-4e9b912358c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" Nov 02 08:51:58.500279 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.500205 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f2cx2\" (UID: \"d27d1869-2b4d-45c9-b878-4e9b912358c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" Nov 02 08:51:58.500739 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.500715 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f2cx2\" (UID: \"d27d1869-2b4d-45c9-b878-4e9b912358c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" Nov 02 08:51:58.502536 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.502513 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f2cx2\" (UID: \"d27d1869-2b4d-45c9-b878-4e9b912358c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" Nov 02 08:51:58.524314 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.524287 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sgcn\" (UniqueName: \"kubernetes.io/projected/d27d1869-2b4d-45c9-b878-4e9b912358c6-kube-api-access-5sgcn\") pod \"maas-api-revoke-sub-3d4156f1-f2cx2\" (UID: \"d27d1869-2b4d-45c9-b878-4e9b912358c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" Nov 02 08:51:58.634353 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.634260 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" Nov 02 08:51:58.934382 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:58.934354 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2"] Nov 02 08:51:58.936707 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:51:58.936678 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27d1869_2b4d_45c9_b878_4e9b912358c6.slice/crio-335da6fb44825e9c4146dc5fa707d3a069a3310684630972212f92750bab09d5 WatchSource:0}: Error finding container 335da6fb44825e9c4146dc5fa707d3a069a3310684630972212f92750bab09d5: Status 404 returned error can't find the container with id 335da6fb44825e9c4146dc5fa707d3a069a3310684630972212f92750bab09d5 Nov 02 08:51:59.340419 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:59.340378 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" event={"ID":"d27d1869-2b4d-45c9-b878-4e9b912358c6","Type":"ContainerStarted","Data":"460d0863bf9a4d314c31e35fffa3b588682ad05d68715b6d07e1faf351ecf62b"} Nov 02 08:51:59.340838 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:59.340446 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" event={"ID":"d27d1869-2b4d-45c9-b878-4e9b912358c6","Type":"ContainerStarted","Data":"335da6fb44825e9c4146dc5fa707d3a069a3310684630972212f92750bab09d5"} Nov 02 08:51:59.391225 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:59.391152 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" podStartSLOduration=1.391137413 podStartE2EDuration="1.391137413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:51:59.389503186 +0000 UTC m=+5219.687784011" watchObservedRunningTime="2026-11-02 08:51:59.391137413 +0000 UTC m=+5219.689418237" Nov 02 08:51:59.825197 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:51:59.825147 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6" path="/var/lib/kubelet/pods/3c7a6b61-34cf-4bd4-bfe3-88890b6a42f6/volumes" Nov 02 08:52:00.347525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:00.347485 5899 generic.go:356] "Generic (PLEG): container finished" podID="d27d1869-2b4d-45c9-b878-4e9b912358c6" containerID="460d0863bf9a4d314c31e35fffa3b588682ad05d68715b6d07e1faf351ecf62b" exitCode=0 Nov 02 08:52:00.348079 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:00.347539 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" event={"ID":"d27d1869-2b4d-45c9-b878-4e9b912358c6","Type":"ContainerDied","Data":"460d0863bf9a4d314c31e35fffa3b588682ad05d68715b6d07e1faf351ecf62b"} Nov 02 08:52:01.493251 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:01.493222 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" Nov 02 08:52:01.535436 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:01.535399 5899 scope.go:122] "RemoveContainer" containerID="260c83b42f799a8a9a3d6cfd1593775fb9c94a6e3a8011228d5f156408a9606f" Nov 02 08:52:01.545712 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:01.545686 5899 scope.go:122] "RemoveContainer" containerID="2cff246cc99d5cee1ce712058faa29a55402d4e7931dc7c5ad0482e7e6d42760" Nov 02 08:52:01.636836 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:01.636751 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-xks-ca\") pod \"d27d1869-2b4d-45c9-b878-4e9b912358c6\" (UID: \"d27d1869-2b4d-45c9-b878-4e9b912358c6\") " Nov 02 08:52:01.636836 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:01.636824 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27d1869-2b4d-45c9-b878-4e9b912358c6-kube-api-access-5sgcn\" (UniqueName: \"kubernetes.io/projected/d27d1869-2b4d-45c9-b878-4e9b912358c6-kube-api-access-5sgcn\") pod \"d27d1869-2b4d-45c9-b878-4e9b912358c6\" (UID: \"d27d1869-2b4d-45c9-b878-4e9b912358c6\") " Nov 02 08:52:01.637087 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:01.636918 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-openshift-ca\") pod \"d27d1869-2b4d-45c9-b878-4e9b912358c6\" (UID: \"d27d1869-2b4d-45c9-b878-4e9b912358c6\") " Nov 02 08:52:01.637281 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:01.637256 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-openshift-ca" pod "d27d1869-2b4d-45c9-b878-4e9b912358c6" (UID: "d27d1869-2b4d-45c9-b878-4e9b912358c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:52:01.638992 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:01.638970 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-xks-ca" pod "d27d1869-2b4d-45c9-b878-4e9b912358c6" (UID: "d27d1869-2b4d-45c9-b878-4e9b912358c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:52:01.639074 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:01.638974 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27d1869-2b4d-45c9-b878-4e9b912358c6-kube-api-access-5sgcn" pod "d27d1869-2b4d-45c9-b878-4e9b912358c6" (UID: "d27d1869-2b4d-45c9-b878-4e9b912358c6"). InnerVolumeSpecName "kube-api-access-5sgcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:52:01.738275 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:01.738233 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:52:01.738275 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:01.738267 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27d1869-2b4d-45c9-b878-4e9b912358c6-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:52:01.738275 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:01.738278 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sgcn\" (UniqueName: \"kubernetes.io/projected/d27d1869-2b4d-45c9-b878-4e9b912358c6-kube-api-access-5sgcn\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:52:02.359593 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:02.359560 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" Nov 02 08:52:02.359790 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:02.359599 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2" event={"ID":"d27d1869-2b4d-45c9-b878-4e9b912358c6","Type":"ContainerDied","Data":"335da6fb44825e9c4146dc5fa707d3a069a3310684630972212f92750bab09d5"} Nov 02 08:52:02.359790 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:02.359652 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="335da6fb44825e9c4146dc5fa707d3a069a3310684630972212f92750bab09d5" Nov 02 08:52:12.034276 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:12.034187 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:52:12.195555 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:12.195512 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:52:13.141416 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:52:13.141370 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:52:28.793298 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.793262 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7"] Nov 02 08:52:28.793811 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.793702 5899 cpu_manager.go:414] "containerMap: removing container" podUID="d27d1869-2b4d-45c9-b878-4e9b912358c6" containerName="revoke-keys" Nov 02 08:52:28.793811 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.793714 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27d1869-2b4d-45c9-b878-4e9b912358c6" containerName="revoke-keys" Nov 02 08:52:28.793811 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.793782 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27d1869-2b4d-45c9-b878-4e9b912358c6" containerName="revoke-keys" Nov 02 08:52:28.796638 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.796618 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" Nov 02 08:52:28.817662 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.817624 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7"] Nov 02 08:52:28.827515 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.827484 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:52:28.827692 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.827644 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:52:28.854616 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.854581 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc8sg\" (UniqueName: \"kubernetes.io/projected/9da64d6d-8df4-451f-a617-1fa6ad8379a4-kube-api-access-pc8sg\") pod \"maas-api-revoke-sub-e2fc9c5b-22zz7\" (UID: \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" Nov 02 08:52:28.854824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.854631 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-22zz7\" (UID: \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" Nov 02 08:52:28.854824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.854666 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-22zz7\" (UID: \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" Nov 02 08:52:28.956369 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.956324 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8sg\" (UniqueName: \"kubernetes.io/projected/9da64d6d-8df4-451f-a617-1fa6ad8379a4-kube-api-access-pc8sg\") pod \"maas-api-revoke-sub-e2fc9c5b-22zz7\" (UID: \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" Nov 02 08:52:28.956369 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.956374 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-22zz7\" (UID: \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" Nov 02 08:52:28.956664 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.956405 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-22zz7\" (UID: \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" Nov 02 08:52:28.957080 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.957057 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-22zz7\" (UID: \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" Nov 02 08:52:28.958688 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.958659 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-22zz7\" (UID: \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" Nov 02 08:52:28.981238 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:28.981202 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc8sg\" (UniqueName: \"kubernetes.io/projected/9da64d6d-8df4-451f-a617-1fa6ad8379a4-kube-api-access-pc8sg\") pod \"maas-api-revoke-sub-e2fc9c5b-22zz7\" (UID: \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" Nov 02 08:52:29.111235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:29.111114 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" Nov 02 08:52:29.411852 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:29.411825 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7"] Nov 02 08:52:29.414260 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:52:29.414233 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da64d6d_8df4_451f_a617_1fa6ad8379a4.slice/crio-cb86f93a19b51d6daa63e172fb4441715c3f5dae5018117630aa8e1ff7a9653a WatchSource:0}: Error finding container cb86f93a19b51d6daa63e172fb4441715c3f5dae5018117630aa8e1ff7a9653a: Status 404 returned error can't find the container with id cb86f93a19b51d6daa63e172fb4441715c3f5dae5018117630aa8e1ff7a9653a Nov 02 08:52:29.515412 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:29.515368 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" event={"ID":"9da64d6d-8df4-451f-a617-1fa6ad8379a4","Type":"ContainerStarted","Data":"cb86f93a19b51d6daa63e172fb4441715c3f5dae5018117630aa8e1ff7a9653a"} Nov 02 08:52:30.522631 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:30.522593 5899 generic.go:356] "Generic (PLEG): container finished" podID="9da64d6d-8df4-451f-a617-1fa6ad8379a4" containerID="7b72dafd43786a0620ef0d405e712958060c97f1098a2f18913c0212c892f5b6" exitCode=0 Nov 02 08:52:30.523042 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:30.522681 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" event={"ID":"9da64d6d-8df4-451f-a617-1fa6ad8379a4","Type":"ContainerDied","Data":"7b72dafd43786a0620ef0d405e712958060c97f1098a2f18913c0212c892f5b6"} Nov 02 08:52:31.659933 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:31.659903 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" Nov 02 08:52:31.785503 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:31.785411 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-openshift-ca\") pod \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\" (UID: \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\") " Nov 02 08:52:31.785503 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:31.785458 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da64d6d-8df4-451f-a617-1fa6ad8379a4-kube-api-access-pc8sg\" (UniqueName: \"kubernetes.io/projected/9da64d6d-8df4-451f-a617-1fa6ad8379a4-kube-api-access-pc8sg\") pod \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\" (UID: \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\") " Nov 02 08:52:31.785754 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:31.785526 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-xks-ca\") pod \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\" (UID: \"9da64d6d-8df4-451f-a617-1fa6ad8379a4\") " Nov 02 08:52:31.785823 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:31.785796 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-openshift-ca" pod "9da64d6d-8df4-451f-a617-1fa6ad8379a4" (UID: "9da64d6d-8df4-451f-a617-1fa6ad8379a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:52:31.785924 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:31.785907 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:52:31.787613 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:31.787588 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da64d6d-8df4-451f-a617-1fa6ad8379a4-kube-api-access-pc8sg" pod "9da64d6d-8df4-451f-a617-1fa6ad8379a4" (UID: "9da64d6d-8df4-451f-a617-1fa6ad8379a4"). InnerVolumeSpecName "kube-api-access-pc8sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:52:31.787736 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:31.787643 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-xks-ca" pod "9da64d6d-8df4-451f-a617-1fa6ad8379a4" (UID: "9da64d6d-8df4-451f-a617-1fa6ad8379a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:52:31.886791 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:31.886738 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc8sg\" (UniqueName: \"kubernetes.io/projected/9da64d6d-8df4-451f-a617-1fa6ad8379a4-kube-api-access-pc8sg\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:52:31.886791 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:31.886780 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da64d6d-8df4-451f-a617-1fa6ad8379a4-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:52:32.534768 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:32.534726 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" Nov 02 08:52:32.534768 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:32.534757 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7" event={"ID":"9da64d6d-8df4-451f-a617-1fa6ad8379a4","Type":"ContainerDied","Data":"cb86f93a19b51d6daa63e172fb4441715c3f5dae5018117630aa8e1ff7a9653a"} Nov 02 08:52:32.535003 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:32.534799 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb86f93a19b51d6daa63e172fb4441715c3f5dae5018117630aa8e1ff7a9653a" Nov 02 08:52:39.158555 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.158514 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz"] Nov 02 08:52:39.159002 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.158947 5899 cpu_manager.go:414] "containerMap: removing container" podUID="9da64d6d-8df4-451f-a617-1fa6ad8379a4" containerName="revoke-keys" Nov 02 08:52:39.159002 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.158959 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da64d6d-8df4-451f-a617-1fa6ad8379a4" containerName="revoke-keys" Nov 02 08:52:39.159098 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.159054 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da64d6d-8df4-451f-a617-1fa6ad8379a4" containerName="revoke-keys" Nov 02 08:52:39.162751 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.162718 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" Nov 02 08:52:39.179356 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.179327 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:52:39.192338 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.192309 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:52:39.194508 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.194483 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz"] Nov 02 08:52:39.258591 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.258552 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-2c4kz\" (UID: \"71dcd9a8-5475-4d16-a972-eec434a90f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" Nov 02 08:52:39.258750 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.258686 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-2c4kz\" (UID: \"71dcd9a8-5475-4d16-a972-eec434a90f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" Nov 02 08:52:39.258750 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.258729 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76nlp\" (UniqueName: \"kubernetes.io/projected/71dcd9a8-5475-4d16-a972-eec434a90f66-kube-api-access-76nlp\") pod \"maas-api-revoke-sub-5a29efb3-2c4kz\" (UID: \"71dcd9a8-5475-4d16-a972-eec434a90f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" Nov 02 08:52:39.359464 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.359424 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-2c4kz\" (UID: \"71dcd9a8-5475-4d16-a972-eec434a90f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" Nov 02 08:52:39.359464 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.359470 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76nlp\" (UniqueName: \"kubernetes.io/projected/71dcd9a8-5475-4d16-a972-eec434a90f66-kube-api-access-76nlp\") pod \"maas-api-revoke-sub-5a29efb3-2c4kz\" (UID: \"71dcd9a8-5475-4d16-a972-eec434a90f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" Nov 02 08:52:39.359670 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.359520 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-2c4kz\" (UID: \"71dcd9a8-5475-4d16-a972-eec434a90f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" Nov 02 08:52:39.360243 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.360223 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-2c4kz\" (UID: \"71dcd9a8-5475-4d16-a972-eec434a90f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" Nov 02 08:52:39.361815 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.361793 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-2c4kz\" (UID: \"71dcd9a8-5475-4d16-a972-eec434a90f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" Nov 02 08:52:39.383878 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.383848 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76nlp\" (UniqueName: \"kubernetes.io/projected/71dcd9a8-5475-4d16-a972-eec434a90f66-kube-api-access-76nlp\") pod \"maas-api-revoke-sub-5a29efb3-2c4kz\" (UID: \"71dcd9a8-5475-4d16-a972-eec434a90f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" Nov 02 08:52:39.476037 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.475953 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" Nov 02 08:52:39.775507 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:39.775480 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz"] Nov 02 08:52:39.777835 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:52:39.777807 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71dcd9a8_5475_4d16_a972_eec434a90f66.slice/crio-1af2eb93f5d30994be3279f5de1f8dc3f45c39286eb0d71370ada4adca3a3cf6 WatchSource:0}: Error finding container 1af2eb93f5d30994be3279f5de1f8dc3f45c39286eb0d71370ada4adca3a3cf6: Status 404 returned error can't find the container with id 1af2eb93f5d30994be3279f5de1f8dc3f45c39286eb0d71370ada4adca3a3cf6 Nov 02 08:52:40.580515 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:40.579466 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" event={"ID":"71dcd9a8-5475-4d16-a972-eec434a90f66","Type":"ContainerStarted","Data":"7b57ad3ef5d38fc731756cbd4ea905236d2c8b7cbba08fd19176b4e2ed945ed3"} Nov 02 08:52:40.580515 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:40.579596 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" event={"ID":"71dcd9a8-5475-4d16-a972-eec434a90f66","Type":"ContainerStarted","Data":"1af2eb93f5d30994be3279f5de1f8dc3f45c39286eb0d71370ada4adca3a3cf6"} Nov 02 08:52:40.623528 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:40.623448 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" podStartSLOduration=1.623429299 podStartE2EDuration="1.623429299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:52:40.622191985 +0000 UTC m=+5260.920472802" watchObservedRunningTime="2026-11-02 08:52:40.623429299 +0000 UTC m=+5260.921710125" Nov 02 08:52:41.613960 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:41.613916 5899 generic.go:356] "Generic (PLEG): container finished" podID="71dcd9a8-5475-4d16-a972-eec434a90f66" containerID="7b57ad3ef5d38fc731756cbd4ea905236d2c8b7cbba08fd19176b4e2ed945ed3" exitCode=0 Nov 02 08:52:41.626186 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:41.614699 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" event={"ID":"71dcd9a8-5475-4d16-a972-eec434a90f66","Type":"ContainerDied","Data":"7b57ad3ef5d38fc731756cbd4ea905236d2c8b7cbba08fd19176b4e2ed945ed3"} Nov 02 08:52:42.965536 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:42.965228 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" Nov 02 08:52:43.049984 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:43.049945 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-xks-ca\") pod \"71dcd9a8-5475-4d16-a972-eec434a90f66\" (UID: \"71dcd9a8-5475-4d16-a972-eec434a90f66\") " Nov 02 08:52:43.050197 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:43.050058 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-openshift-ca\") pod \"71dcd9a8-5475-4d16-a972-eec434a90f66\" (UID: \"71dcd9a8-5475-4d16-a972-eec434a90f66\") " Nov 02 08:52:43.050197 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:43.050150 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71dcd9a8-5475-4d16-a972-eec434a90f66-kube-api-access-76nlp\" (UniqueName: \"kubernetes.io/projected/71dcd9a8-5475-4d16-a972-eec434a90f66-kube-api-access-76nlp\") pod \"71dcd9a8-5475-4d16-a972-eec434a90f66\" (UID: \"71dcd9a8-5475-4d16-a972-eec434a90f66\") " Nov 02 08:52:43.050563 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:43.050528 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-openshift-ca" pod "71dcd9a8-5475-4d16-a972-eec434a90f66" (UID: "71dcd9a8-5475-4d16-a972-eec434a90f66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:52:43.052790 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:43.052759 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-xks-ca" pod "71dcd9a8-5475-4d16-a972-eec434a90f66" (UID: "71dcd9a8-5475-4d16-a972-eec434a90f66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:52:43.052953 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:43.052926 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dcd9a8-5475-4d16-a972-eec434a90f66-kube-api-access-76nlp" pod "71dcd9a8-5475-4d16-a972-eec434a90f66" (UID: "71dcd9a8-5475-4d16-a972-eec434a90f66"). InnerVolumeSpecName "kube-api-access-76nlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:52:43.143504 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:52:43.143460 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:52:43.151030 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:43.151010 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:52:43.151106 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:43.151033 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dcd9a8-5475-4d16-a972-eec434a90f66-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:52:43.151106 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:43.151043 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76nlp\" (UniqueName: \"kubernetes.io/projected/71dcd9a8-5475-4d16-a972-eec434a90f66-kube-api-access-76nlp\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:52:43.636936 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:43.636900 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" Nov 02 08:52:43.637119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:43.636931 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz" event={"ID":"71dcd9a8-5475-4d16-a972-eec434a90f66","Type":"ContainerDied","Data":"1af2eb93f5d30994be3279f5de1f8dc3f45c39286eb0d71370ada4adca3a3cf6"} Nov 02 08:52:43.637119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:43.636979 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1af2eb93f5d30994be3279f5de1f8dc3f45c39286eb0d71370ada4adca3a3cf6" Nov 02 08:52:54.966370 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:54.966332 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2"] Nov 02 08:52:54.979507 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:54.979469 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f2cx2"] Nov 02 08:52:55.019621 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.019587 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq"] Nov 02 08:52:55.020066 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.020051 5899 cpu_manager.go:414] "containerMap: removing container" podUID="71dcd9a8-5475-4d16-a972-eec434a90f66" containerName="revoke-keys" Nov 02 08:52:55.020115 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.020068 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71dcd9a8-5475-4d16-a972-eec434a90f66" containerName="revoke-keys" Nov 02 08:52:55.020188 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.020178 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="71dcd9a8-5475-4d16-a972-eec434a90f66" containerName="revoke-keys" Nov 02 08:52:55.023355 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.023334 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" Nov 02 08:52:55.040579 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.040550 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:52:55.040809 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.040788 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:52:55.043909 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.043852 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq"] Nov 02 08:52:55.069715 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.069673 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md8lp\" (UniqueName: \"kubernetes.io/projected/53187f45-10c6-4f3f-966c-142c65f89739-kube-api-access-md8lp\") pod \"maas-api-revoke-sub-3d4156f1-6bjfq\" (UID: \"53187f45-10c6-4f3f-966c-142c65f89739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" Nov 02 08:52:55.069901 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.069765 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6bjfq\" (UID: \"53187f45-10c6-4f3f-966c-142c65f89739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" Nov 02 08:52:55.069901 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.069815 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6bjfq\" (UID: \"53187f45-10c6-4f3f-966c-142c65f89739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" Nov 02 08:52:55.170566 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.170513 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6bjfq\" (UID: \"53187f45-10c6-4f3f-966c-142c65f89739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" Nov 02 08:52:55.170785 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.170630 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md8lp\" (UniqueName: \"kubernetes.io/projected/53187f45-10c6-4f3f-966c-142c65f89739-kube-api-access-md8lp\") pod \"maas-api-revoke-sub-3d4156f1-6bjfq\" (UID: \"53187f45-10c6-4f3f-966c-142c65f89739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" Nov 02 08:52:55.170785 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.170699 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6bjfq\" (UID: \"53187f45-10c6-4f3f-966c-142c65f89739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" Nov 02 08:52:55.171402 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.171380 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6bjfq\" (UID: \"53187f45-10c6-4f3f-966c-142c65f89739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" Nov 02 08:52:55.172870 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.172842 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6bjfq\" (UID: \"53187f45-10c6-4f3f-966c-142c65f89739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" Nov 02 08:52:55.194678 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.194654 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md8lp\" (UniqueName: \"kubernetes.io/projected/53187f45-10c6-4f3f-966c-142c65f89739-kube-api-access-md8lp\") pod \"maas-api-revoke-sub-3d4156f1-6bjfq\" (UID: \"53187f45-10c6-4f3f-966c-142c65f89739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" Nov 02 08:52:55.337043 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.337005 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" Nov 02 08:52:55.656500 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.656471 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq"] Nov 02 08:52:55.658334 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:52:55.658299 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53187f45_10c6_4f3f_966c_142c65f89739.slice/crio-734870de6a591122bc7e25d918e175e20d1822294e28c3cd4c6a82e786f84bfe WatchSource:0}: Error finding container 734870de6a591122bc7e25d918e175e20d1822294e28c3cd4c6a82e786f84bfe: Status 404 returned error can't find the container with id 734870de6a591122bc7e25d918e175e20d1822294e28c3cd4c6a82e786f84bfe Nov 02 08:52:55.701126 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.701095 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" event={"ID":"53187f45-10c6-4f3f-966c-142c65f89739","Type":"ContainerStarted","Data":"734870de6a591122bc7e25d918e175e20d1822294e28c3cd4c6a82e786f84bfe"} Nov 02 08:52:55.819123 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:55.819088 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27d1869-2b4d-45c9-b878-4e9b912358c6" path="/var/lib/kubelet/pods/d27d1869-2b4d-45c9-b878-4e9b912358c6/volumes" Nov 02 08:52:56.708469 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:56.708434 5899 generic.go:356] "Generic (PLEG): container finished" podID="53187f45-10c6-4f3f-966c-142c65f89739" containerID="85c7b74815b6453ef1ef56daaa5f0e8fcaa2befe5ca6d23f2dafceaf96750892" exitCode=0 Nov 02 08:52:56.708865 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:56.708476 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" event={"ID":"53187f45-10c6-4f3f-966c-142c65f89739","Type":"ContainerDied","Data":"85c7b74815b6453ef1ef56daaa5f0e8fcaa2befe5ca6d23f2dafceaf96750892"} Nov 02 08:52:57.856099 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:57.856072 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" Nov 02 08:52:58.001290 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:58.001256 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-xks-ca\") pod \"53187f45-10c6-4f3f-966c-142c65f89739\" (UID: \"53187f45-10c6-4f3f-966c-142c65f89739\") " Nov 02 08:52:58.001491 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:58.001321 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-openshift-ca\") pod \"53187f45-10c6-4f3f-966c-142c65f89739\" (UID: \"53187f45-10c6-4f3f-966c-142c65f89739\") " Nov 02 08:52:58.001491 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:58.001387 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53187f45-10c6-4f3f-966c-142c65f89739-kube-api-access-md8lp\" (UniqueName: \"kubernetes.io/projected/53187f45-10c6-4f3f-966c-142c65f89739-kube-api-access-md8lp\") pod \"53187f45-10c6-4f3f-966c-142c65f89739\" (UID: \"53187f45-10c6-4f3f-966c-142c65f89739\") " Nov 02 08:52:58.001721 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:58.001693 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-openshift-ca" pod "53187f45-10c6-4f3f-966c-142c65f89739" (UID: "53187f45-10c6-4f3f-966c-142c65f89739"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:52:58.003525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:58.003505 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-xks-ca" pod "53187f45-10c6-4f3f-966c-142c65f89739" (UID: "53187f45-10c6-4f3f-966c-142c65f89739"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:52:58.003637 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:58.003615 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53187f45-10c6-4f3f-966c-142c65f89739-kube-api-access-md8lp" pod "53187f45-10c6-4f3f-966c-142c65f89739" (UID: "53187f45-10c6-4f3f-966c-142c65f89739"). InnerVolumeSpecName "kube-api-access-md8lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:52:58.103070 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:58.103030 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:52:58.103070 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:58.103063 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53187f45-10c6-4f3f-966c-142c65f89739-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:52:58.103070 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:58.103074 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md8lp\" (UniqueName: \"kubernetes.io/projected/53187f45-10c6-4f3f-966c-142c65f89739-kube-api-access-md8lp\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:52:58.720337 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:58.720290 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" event={"ID":"53187f45-10c6-4f3f-966c-142c65f89739","Type":"ContainerDied","Data":"734870de6a591122bc7e25d918e175e20d1822294e28c3cd4c6a82e786f84bfe"} Nov 02 08:52:58.720337 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:58.720332 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq" Nov 02 08:52:58.720631 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:52:58.720355 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="734870de6a591122bc7e25d918e175e20d1822294e28c3cd4c6a82e786f84bfe" Nov 02 08:53:01.578380 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:01.578332 5899 scope.go:122] "RemoveContainer" containerID="bdd8388894faf0ec9dd6f1b75a0409a1c62f183047a891959089fccf45ffae79" Nov 02 08:53:10.200690 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:10.200639 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:10.726919 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:10.726877 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:10.923545 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:10.923512 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:12.160481 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:12.160448 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:12.289049 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:12.289007 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:13.140286 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:53:13.140249 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:53:14.862942 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:14.862894 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:14.996598 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:14.996561 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:16.495902 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:16.495864 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:16.647000 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:16.646965 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:16.837464 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:16.837393 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:18.320746 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:18.320709 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:20.154825 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:20.154784 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:21.928842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:21.928804 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:23.624231 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:23.624187 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:25.426317 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:25.426285 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:27.399739 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:27.399706 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:29.121636 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:29.121600 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:29.725332 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:29.725295 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:30.317709 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:30.317672 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:30.461009 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:30.460956 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:30.621657 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:30.621575 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:31.029848 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:31.029817 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:31.433063 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:31.432979 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:32.807735 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:32.807699 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:34.529018 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:34.528976 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:36.322789 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:36.322756 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:38.349661 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:38.349620 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:40.182887 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:40.182848 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:41.030595 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:41.030517 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp"] Nov 02 08:53:41.043416 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:41.043388 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s2dzp"] Nov 02 08:53:41.818402 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:41.818370 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ec716e-c29f-431b-a58d-1f31cb759ea5" path="/var/lib/kubelet/pods/d2ec716e-c29f-431b-a58d-1f31cb759ea5/volumes" Nov 02 08:53:41.890194 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:41.890140 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:43.140180 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:53:43.140122 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:53:43.676128 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:43.676089 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:45.426890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:45.426843 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:47.099207 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:47.099130 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:48.820480 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:48.820444 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:50.581720 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:50.581665 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:52.433022 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:52.432984 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:54.208514 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:54.208476 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:55.988364 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:55.988313 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:57.634783 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:57.634736 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:53:59.405370 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:53:59.405335 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:01.126765 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:01.126720 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:02.815712 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:02.815675 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:04.374361 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.374324 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q"] Nov 02 08:54:04.374856 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.374769 5899 cpu_manager.go:414] "containerMap: removing container" podUID="53187f45-10c6-4f3f-966c-142c65f89739" containerName="revoke-keys" Nov 02 08:54:04.374856 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.374786 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53187f45-10c6-4f3f-966c-142c65f89739" containerName="revoke-keys" Nov 02 08:54:04.374936 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.374875 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="53187f45-10c6-4f3f-966c-142c65f89739" containerName="revoke-keys" Nov 02 08:54:04.378106 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.378089 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" Nov 02 08:54:04.397972 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.397939 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q"] Nov 02 08:54:04.408257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.408225 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:54:04.408424 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.408327 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:54:04.468328 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.468288 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d0f0d4-s7h2q\" (UID: \"99094ba7-68dd-4e7c-b1a7-61332f95051d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" Nov 02 08:54:04.468530 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.468506 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d0f0d4-s7h2q\" (UID: \"99094ba7-68dd-4e7c-b1a7-61332f95051d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" Nov 02 08:54:04.468605 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.468546 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbw5r\" (UniqueName: \"kubernetes.io/projected/99094ba7-68dd-4e7c-b1a7-61332f95051d-kube-api-access-tbw5r\") pod \"maas-api-revoke-sub-25d0f0d4-s7h2q\" (UID: \"99094ba7-68dd-4e7c-b1a7-61332f95051d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" Nov 02 08:54:04.509411 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.509367 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:04.569842 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.569800 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d0f0d4-s7h2q\" (UID: \"99094ba7-68dd-4e7c-b1a7-61332f95051d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" Nov 02 08:54:04.570048 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.569907 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d0f0d4-s7h2q\" (UID: \"99094ba7-68dd-4e7c-b1a7-61332f95051d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" Nov 02 08:54:04.570048 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.569931 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbw5r\" (UniqueName: \"kubernetes.io/projected/99094ba7-68dd-4e7c-b1a7-61332f95051d-kube-api-access-tbw5r\") pod \"maas-api-revoke-sub-25d0f0d4-s7h2q\" (UID: \"99094ba7-68dd-4e7c-b1a7-61332f95051d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" Nov 02 08:54:04.570648 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.570617 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d0f0d4-s7h2q\" (UID: \"99094ba7-68dd-4e7c-b1a7-61332f95051d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" Nov 02 08:54:04.572409 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.572387 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d0f0d4-s7h2q\" (UID: \"99094ba7-68dd-4e7c-b1a7-61332f95051d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" Nov 02 08:54:04.594474 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.594449 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbw5r\" (UniqueName: \"kubernetes.io/projected/99094ba7-68dd-4e7c-b1a7-61332f95051d-kube-api-access-tbw5r\") pod \"maas-api-revoke-sub-25d0f0d4-s7h2q\" (UID: \"99094ba7-68dd-4e7c-b1a7-61332f95051d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" Nov 02 08:54:04.691144 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.691057 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" Nov 02 08:54:04.991344 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:04.991318 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q"] Nov 02 08:54:04.993617 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:54:04.993588 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99094ba7_68dd_4e7c_b1a7_61332f95051d.slice/crio-7290898b49f63a5d3fa109432f281265f8c0a3d8e094f26fe5505ceef9170234 WatchSource:0}: Error finding container 7290898b49f63a5d3fa109432f281265f8c0a3d8e094f26fe5505ceef9170234: Status 404 returned error can't find the container with id 7290898b49f63a5d3fa109432f281265f8c0a3d8e094f26fe5505ceef9170234 Nov 02 08:54:05.030511 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:05.030481 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s"] Nov 02 08:54:05.034216 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:05.034191 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9hl7s"] Nov 02 08:54:05.143932 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:05.143891 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" event={"ID":"99094ba7-68dd-4e7c-b1a7-61332f95051d","Type":"ContainerStarted","Data":"7290898b49f63a5d3fa109432f281265f8c0a3d8e094f26fe5505ceef9170234"} Nov 02 08:54:05.820262 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:05.820215 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4bfd0e-bd84-4bc8-af11-4c8602afa009" path="/var/lib/kubelet/pods/1c4bfd0e-bd84-4bc8-af11-4c8602afa009/volumes" Nov 02 08:54:06.151068 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:06.150975 5899 generic.go:356] "Generic (PLEG): container finished" podID="99094ba7-68dd-4e7c-b1a7-61332f95051d" containerID="fa801c97a1d7f37aafbd2c3217498ffc3d7ab300be71a53a2da8b927b4e49204" exitCode=0 Nov 02 08:54:06.151283 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:06.151059 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" event={"ID":"99094ba7-68dd-4e7c-b1a7-61332f95051d","Type":"ContainerDied","Data":"fa801c97a1d7f37aafbd2c3217498ffc3d7ab300be71a53a2da8b927b4e49204"} Nov 02 08:54:06.305840 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:06.305806 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:07.290199 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:07.290151 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" Nov 02 08:54:07.397247 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:07.397207 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-openshift-ca\") pod \"99094ba7-68dd-4e7c-b1a7-61332f95051d\" (UID: \"99094ba7-68dd-4e7c-b1a7-61332f95051d\") " Nov 02 08:54:07.397434 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:07.397370 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-xks-ca\") pod \"99094ba7-68dd-4e7c-b1a7-61332f95051d\" (UID: \"99094ba7-68dd-4e7c-b1a7-61332f95051d\") " Nov 02 08:54:07.397484 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:07.397446 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99094ba7-68dd-4e7c-b1a7-61332f95051d-kube-api-access-tbw5r\" (UniqueName: \"kubernetes.io/projected/99094ba7-68dd-4e7c-b1a7-61332f95051d-kube-api-access-tbw5r\") pod \"99094ba7-68dd-4e7c-b1a7-61332f95051d\" (UID: \"99094ba7-68dd-4e7c-b1a7-61332f95051d\") " Nov 02 08:54:07.397675 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:07.397652 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-openshift-ca" pod "99094ba7-68dd-4e7c-b1a7-61332f95051d" (UID: "99094ba7-68dd-4e7c-b1a7-61332f95051d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:54:07.397786 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:07.397758 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:54:07.399536 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:07.399509 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99094ba7-68dd-4e7c-b1a7-61332f95051d-kube-api-access-tbw5r" pod "99094ba7-68dd-4e7c-b1a7-61332f95051d" (UID: "99094ba7-68dd-4e7c-b1a7-61332f95051d"). InnerVolumeSpecName "kube-api-access-tbw5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:54:07.399629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:07.399544 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-xks-ca" pod "99094ba7-68dd-4e7c-b1a7-61332f95051d" (UID: "99094ba7-68dd-4e7c-b1a7-61332f95051d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:54:07.499282 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:07.499245 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbw5r\" (UniqueName: \"kubernetes.io/projected/99094ba7-68dd-4e7c-b1a7-61332f95051d-kube-api-access-tbw5r\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:54:07.499282 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:07.499277 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99094ba7-68dd-4e7c-b1a7-61332f95051d-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:54:08.007823 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:08.007790 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:08.163520 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:08.163438 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" Nov 02 08:54:08.163520 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:08.163476 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q" event={"ID":"99094ba7-68dd-4e7c-b1a7-61332f95051d","Type":"ContainerDied","Data":"7290898b49f63a5d3fa109432f281265f8c0a3d8e094f26fe5505ceef9170234"} Nov 02 08:54:08.163716 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:08.163526 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7290898b49f63a5d3fa109432f281265f8c0a3d8e094f26fe5505ceef9170234" Nov 02 08:54:09.814091 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:09.814055 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:11.670120 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:11.670012 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:12.294031 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:12.293994 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:13.140302 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:54:13.140267 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:54:13.409207 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:13.409093 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:15.105335 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:15.105304 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:15.413428 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:15.413339 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:15.888712 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:15.888680 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:16.812263 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:16.812225 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:18.608909 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:18.608873 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:20.304442 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:20.304403 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:20.518929 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:20.518889 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:21.455339 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:21.455307 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:21.585077 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:21.585043 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:22.097391 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:22.097359 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:23.065432 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:23.065399 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:23.707557 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:23.707507 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:24.032629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:24.032596 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc"] Nov 02 08:54:24.045367 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:24.045337 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wl4xc"] Nov 02 08:54:25.818301 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:25.818267 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a6a5c24-c1c8-42e0-9638-53c9c7f787f0" path="/var/lib/kubelet/pods/2a6a5c24-c1c8-42e0-9638-53c9c7f787f0/volumes" Nov 02 08:54:28.744899 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:28.744857 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:28.890665 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:28.890623 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:36.028297 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:36.028224 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq"] Nov 02 08:54:36.040768 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:36.040734 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-24nxq"] Nov 02 08:54:37.818032 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:37.817995 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="028899e5-16ed-4a11-b932-f5264ce2bae9" path="/var/lib/kubelet/pods/028899e5-16ed-4a11-b932-f5264ce2bae9/volumes" Nov 02 08:54:43.143416 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:54:43.143372 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:54:51.376895 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:51.376849 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:54.426221 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:54.426186 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:54.580095 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:54.580051 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:55.794797 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:55.794761 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:55.938441 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:55.938407 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:56.114952 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:56.114870 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:56.290972 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:56.290935 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:57.605658 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:57.605620 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:54:59.384417 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:54:59.384378 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:01.096069 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:01.096035 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:01.608053 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:01.608017 5899 scope.go:122] "RemoveContainer" containerID="1c273cbb68cac137f3392116bed6c39d163bae65b02693f45f58f94307733869" Nov 02 08:55:01.619246 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:01.619213 5899 scope.go:122] "RemoveContainer" containerID="e8a2b2f8c17985330cac81604428962ce9b8547e31271a4b6d080dfdc8b49ce6" Nov 02 08:55:02.930692 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:02.930660 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:03.034934 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:03.034896 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h"] Nov 02 08:55:03.047442 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:03.047402 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-c667h"] Nov 02 08:55:03.366950 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:03.366913 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:03.818555 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:03.818524 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d905e0-7907-473d-b7f6-f926adc0305c" path="/var/lib/kubelet/pods/d2d905e0-7907-473d-b7f6-f926adc0305c/volumes" Nov 02 08:55:03.849619 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:03.849588 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:04.404479 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.404440 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm"] Nov 02 08:55:04.404870 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.404857 5899 cpu_manager.go:414] "containerMap: removing container" podUID="99094ba7-68dd-4e7c-b1a7-61332f95051d" containerName="revoke-keys" Nov 02 08:55:04.404920 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.404871 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99094ba7-68dd-4e7c-b1a7-61332f95051d" containerName="revoke-keys" Nov 02 08:55:04.404970 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.404960 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="99094ba7-68dd-4e7c-b1a7-61332f95051d" containerName="revoke-keys" Nov 02 08:55:04.409891 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.409869 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" Nov 02 08:55:04.425985 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.425959 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:55:04.439240 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.439218 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:55:04.442607 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.442577 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm"] Nov 02 08:55:04.550890 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.550856 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm\" (UID: \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" Nov 02 08:55:04.551063 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.550895 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm\" (UID: \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" Nov 02 08:55:04.551114 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.551051 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtfx\" (UniqueName: \"kubernetes.io/projected/d75779cf-ff9c-40a1-8f98-af1a3b381c23-kube-api-access-7qtfx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm\" (UID: \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" Nov 02 08:55:04.652076 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.652038 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm\" (UID: \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" Nov 02 08:55:04.652076 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.652079 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm\" (UID: \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" Nov 02 08:55:04.652373 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.652133 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtfx\" (UniqueName: \"kubernetes.io/projected/d75779cf-ff9c-40a1-8f98-af1a3b381c23-kube-api-access-7qtfx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm\" (UID: \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" Nov 02 08:55:04.652793 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.652751 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm\" (UID: \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" Nov 02 08:55:04.654515 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.654468 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm\" (UID: \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" Nov 02 08:55:04.677670 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.677647 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtfx\" (UniqueName: \"kubernetes.io/projected/d75779cf-ff9c-40a1-8f98-af1a3b381c23-kube-api-access-7qtfx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm\" (UID: \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" Nov 02 08:55:04.724538 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.724503 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" Nov 02 08:55:04.934449 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:04.934347 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:05.023771 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:05.023740 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm"] Nov 02 08:55:05.026521 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:55:05.026490 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75779cf_ff9c_40a1_8f98_af1a3b381c23.slice/crio-bf9fb3ade211c36fe1fa80d706ab8fdf635abf5c9bd8ba51273e537049d2ee4c WatchSource:0}: Error finding container bf9fb3ade211c36fe1fa80d706ab8fdf635abf5c9bd8ba51273e537049d2ee4c: Status 404 returned error can't find the container with id bf9fb3ade211c36fe1fa80d706ab8fdf635abf5c9bd8ba51273e537049d2ee4c Nov 02 08:55:05.476501 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:05.476461 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" event={"ID":"d75779cf-ff9c-40a1-8f98-af1a3b381c23","Type":"ContainerStarted","Data":"b504738262590d625877f11ba7f0fc57f31fb389f213013860f52bff3c2042cf"} Nov 02 08:55:05.476907 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:05.476528 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" event={"ID":"d75779cf-ff9c-40a1-8f98-af1a3b381c23","Type":"ContainerStarted","Data":"bf9fb3ade211c36fe1fa80d706ab8fdf635abf5c9bd8ba51273e537049d2ee4c"} Nov 02 08:55:05.518537 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:05.518472 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" podStartSLOduration=1.518458679 podStartE2EDuration="1.518458679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:55:05.5176358 +0000 UTC m=+5405.815916625" watchObservedRunningTime="2026-11-02 08:55:05.518458679 +0000 UTC m=+5405.816739504" Nov 02 08:55:06.483249 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:06.483214 5899 generic.go:356] "Generic (PLEG): container finished" podID="d75779cf-ff9c-40a1-8f98-af1a3b381c23" containerID="b504738262590d625877f11ba7f0fc57f31fb389f213013860f52bff3c2042cf" exitCode=0 Nov 02 08:55:06.483662 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:06.483303 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" event={"ID":"d75779cf-ff9c-40a1-8f98-af1a3b381c23","Type":"ContainerDied","Data":"b504738262590d625877f11ba7f0fc57f31fb389f213013860f52bff3c2042cf"} Nov 02 08:55:06.663963 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:06.663895 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:07.626869 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:07.626844 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" Nov 02 08:55:07.784813 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:07.784729 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-xks-ca\") pod \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\" (UID: \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\") " Nov 02 08:55:07.784813 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:07.784775 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-openshift-ca\") pod \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\" (UID: \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\") " Nov 02 08:55:07.784813 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:07.784811 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75779cf-ff9c-40a1-8f98-af1a3b381c23-kube-api-access-7qtfx\" (UniqueName: \"kubernetes.io/projected/d75779cf-ff9c-40a1-8f98-af1a3b381c23-kube-api-access-7qtfx\") pod \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\" (UID: \"d75779cf-ff9c-40a1-8f98-af1a3b381c23\") " Nov 02 08:55:07.785252 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:07.785227 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-openshift-ca" pod "d75779cf-ff9c-40a1-8f98-af1a3b381c23" (UID: "d75779cf-ff9c-40a1-8f98-af1a3b381c23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:55:07.786872 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:07.786839 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-xks-ca" pod "d75779cf-ff9c-40a1-8f98-af1a3b381c23" (UID: "d75779cf-ff9c-40a1-8f98-af1a3b381c23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:55:07.787055 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:07.787034 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75779cf-ff9c-40a1-8f98-af1a3b381c23-kube-api-access-7qtfx" pod "d75779cf-ff9c-40a1-8f98-af1a3b381c23" (UID: "d75779cf-ff9c-40a1-8f98-af1a3b381c23"). InnerVolumeSpecName "kube-api-access-7qtfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:55:07.885972 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:07.885929 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:55:07.885972 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:07.885960 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d75779cf-ff9c-40a1-8f98-af1a3b381c23-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:55:07.885972 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:07.885970 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qtfx\" (UniqueName: \"kubernetes.io/projected/d75779cf-ff9c-40a1-8f98-af1a3b381c23-kube-api-access-7qtfx\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:55:08.422610 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:08.422577 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:08.495564 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:08.495508 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" event={"ID":"d75779cf-ff9c-40a1-8f98-af1a3b381c23","Type":"ContainerDied","Data":"bf9fb3ade211c36fe1fa80d706ab8fdf635abf5c9bd8ba51273e537049d2ee4c"} Nov 02 08:55:08.495771 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:08.495591 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf9fb3ade211c36fe1fa80d706ab8fdf635abf5c9bd8ba51273e537049d2ee4c" Nov 02 08:55:08.495771 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:08.495534 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm" Nov 02 08:55:09.805594 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:09.805559 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:09.949219 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:09.949157 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:10.270571 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:10.270534 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:11.895674 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:11.895609 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:12.420923 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:12.420876 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:13.139978 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:55:13.139941 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:55:13.604424 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:13.604381 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:14.030026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:14.029988 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5"] Nov 02 08:55:14.043007 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:14.042974 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qvbs5"] Nov 02 08:55:15.155885 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:15.155844 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:15.301754 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:15.301719 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:15.431135 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:15.431042 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:15.818536 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:15.818500 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="088d752b-fb3d-49cc-906e-164c6c1402ad" path="/var/lib/kubelet/pods/088d752b-fb3d-49cc-906e-164c6c1402ad/volumes" Nov 02 08:55:16.770903 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:16.770865 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:17.304479 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:17.304438 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:17.438326 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:17.438290 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm"] Nov 02 08:55:17.443888 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:17.443853 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:17.451110 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:17.451075 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5c7145-vrdmm"] Nov 02 08:55:17.818235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:17.818202 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75779cf-ff9c-40a1-8f98-af1a3b381c23" path="/var/lib/kubelet/pods/d75779cf-ff9c-40a1-8f98-af1a3b381c23/volumes" Nov 02 08:55:17.972142 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:17.972104 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:19.203982 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.203945 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq"] Nov 02 08:55:19.216829 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.216799 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6bjfq"] Nov 02 08:55:19.252119 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.252082 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6"] Nov 02 08:55:19.252608 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.252588 5899 cpu_manager.go:414] "containerMap: removing container" podUID="d75779cf-ff9c-40a1-8f98-af1a3b381c23" containerName="revoke-keys" Nov 02 08:55:19.252702 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.252610 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75779cf-ff9c-40a1-8f98-af1a3b381c23" containerName="revoke-keys" Nov 02 08:55:19.252766 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.252715 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75779cf-ff9c-40a1-8f98-af1a3b381c23" containerName="revoke-keys" Nov 02 08:55:19.255950 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.255928 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" Nov 02 08:55:19.273819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.273780 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:55:19.273819 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.273784 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:55:19.275936 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.275907 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6"] Nov 02 08:55:19.294256 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.294225 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wksq6\" (UID: \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" Nov 02 08:55:19.294415 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.294265 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wksq6\" (UID: \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" Nov 02 08:55:19.294501 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.294417 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv9k8\" (UniqueName: \"kubernetes.io/projected/5aaa8786-c58b-461f-9e3d-baccc44b17a6-kube-api-access-cv9k8\") pod \"maas-api-revoke-sub-3d4156f1-wksq6\" (UID: \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" Nov 02 08:55:19.395948 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.395903 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wksq6\" (UID: \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" Nov 02 08:55:19.396142 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.395974 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wksq6\" (UID: \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" Nov 02 08:55:19.396142 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.396133 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9k8\" (UniqueName: \"kubernetes.io/projected/5aaa8786-c58b-461f-9e3d-baccc44b17a6-kube-api-access-cv9k8\") pod \"maas-api-revoke-sub-3d4156f1-wksq6\" (UID: \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" Nov 02 08:55:19.396808 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.396777 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wksq6\" (UID: \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" Nov 02 08:55:19.398736 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.398710 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wksq6\" (UID: \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" Nov 02 08:55:19.445067 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.445038 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv9k8\" (UniqueName: \"kubernetes.io/projected/5aaa8786-c58b-461f-9e3d-baccc44b17a6-kube-api-access-cv9k8\") pod \"maas-api-revoke-sub-3d4156f1-wksq6\" (UID: \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" Nov 02 08:55:19.568629 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.568596 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" Nov 02 08:55:19.818394 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.818360 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53187f45-10c6-4f3f-966c-142c65f89739" path="/var/lib/kubelet/pods/53187f45-10c6-4f3f-966c-142c65f89739/volumes" Nov 02 08:55:19.865739 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:19.865637 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6"] Nov 02 08:55:19.868743 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:55:19.868716 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aaa8786_c58b_461f_9e3d_baccc44b17a6.slice/crio-741d6464bb67fbaf7013edc795751a3a672a24012f4e9bb03a32b8eb6cd89145 WatchSource:0}: Error finding container 741d6464bb67fbaf7013edc795751a3a672a24012f4e9bb03a32b8eb6cd89145: Status 404 returned error can't find the container with id 741d6464bb67fbaf7013edc795751a3a672a24012f4e9bb03a32b8eb6cd89145 Nov 02 08:55:20.565854 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:20.565810 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" event={"ID":"5aaa8786-c58b-461f-9e3d-baccc44b17a6","Type":"ContainerStarted","Data":"13329a38d2c0a5def6a1ab10939de15b93e402d4434b202429b47b4036eb72b5"} Nov 02 08:55:20.566274 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:20.565872 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" event={"ID":"5aaa8786-c58b-461f-9e3d-baccc44b17a6","Type":"ContainerStarted","Data":"741d6464bb67fbaf7013edc795751a3a672a24012f4e9bb03a32b8eb6cd89145"} Nov 02 08:55:20.608936 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:20.608888 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" podStartSLOduration=1.608874631 podStartE2EDuration="1.608874631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:55:20.607659418 +0000 UTC m=+5420.905940249" watchObservedRunningTime="2026-11-02 08:55:20.608874631 +0000 UTC m=+5420.907155455" Nov 02 08:55:21.572688 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:21.572648 5899 generic.go:356] "Generic (PLEG): container finished" podID="5aaa8786-c58b-461f-9e3d-baccc44b17a6" containerID="13329a38d2c0a5def6a1ab10939de15b93e402d4434b202429b47b4036eb72b5" exitCode=0 Nov 02 08:55:21.572688 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:21.572691 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" event={"ID":"5aaa8786-c58b-461f-9e3d-baccc44b17a6","Type":"ContainerDied","Data":"13329a38d2c0a5def6a1ab10939de15b93e402d4434b202429b47b4036eb72b5"} Nov 02 08:55:22.712525 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:22.712496 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" Nov 02 08:55:22.728780 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:22.728743 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-xks-ca\") pod \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\" (UID: \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\") " Nov 02 08:55:22.728780 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:22.728781 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-openshift-ca\") pod \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\" (UID: \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\") " Nov 02 08:55:22.729016 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:22.728876 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aaa8786-c58b-461f-9e3d-baccc44b17a6-kube-api-access-cv9k8\" (UniqueName: \"kubernetes.io/projected/5aaa8786-c58b-461f-9e3d-baccc44b17a6-kube-api-access-cv9k8\") pod \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\" (UID: \"5aaa8786-c58b-461f-9e3d-baccc44b17a6\") " Nov 02 08:55:22.729130 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:22.729106 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-openshift-ca" pod "5aaa8786-c58b-461f-9e3d-baccc44b17a6" (UID: "5aaa8786-c58b-461f-9e3d-baccc44b17a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:55:22.730804 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:22.730780 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-xks-ca" pod "5aaa8786-c58b-461f-9e3d-baccc44b17a6" (UID: "5aaa8786-c58b-461f-9e3d-baccc44b17a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:55:22.730931 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:22.730914 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aaa8786-c58b-461f-9e3d-baccc44b17a6-kube-api-access-cv9k8" pod "5aaa8786-c58b-461f-9e3d-baccc44b17a6" (UID: "5aaa8786-c58b-461f-9e3d-baccc44b17a6"). InnerVolumeSpecName "kube-api-access-cv9k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:55:22.829982 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:22.829897 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:55:22.829982 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:22.829932 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aaa8786-c58b-461f-9e3d-baccc44b17a6-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:55:22.829982 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:22.829942 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv9k8\" (UniqueName: \"kubernetes.io/projected/5aaa8786-c58b-461f-9e3d-baccc44b17a6-kube-api-access-cv9k8\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:55:23.584501 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:23.584468 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" Nov 02 08:55:23.584682 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:23.584510 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6" event={"ID":"5aaa8786-c58b-461f-9e3d-baccc44b17a6","Type":"ContainerDied","Data":"741d6464bb67fbaf7013edc795751a3a672a24012f4e9bb03a32b8eb6cd89145"} Nov 02 08:55:23.584682 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:23.584585 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="741d6464bb67fbaf7013edc795751a3a672a24012f4e9bb03a32b8eb6cd89145" Nov 02 08:55:35.027527 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:35.027485 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j"] Nov 02 08:55:35.040528 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:35.040497 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-pbw6j"] Nov 02 08:55:35.817702 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:35.817667 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ca6a5c-d247-43aa-afb1-15a0bcb985f1" path="/var/lib/kubelet/pods/76ca6a5c-d247-43aa-afb1-15a0bcb985f1/volumes" Nov 02 08:55:40.920882 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:40.920849 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:41.062845 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:55:41.062805 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:55:43.140786 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:55:43.140742 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:56:01.644710 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:56:01.644674 5899 scope.go:122] "RemoveContainer" containerID="64ffbedb0d3356be7d0ad45e4fa1ab3ebd2ee5819b292a822cf0cf4f513bc81d" Nov 02 08:56:01.653697 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:56:01.653676 5899 scope.go:122] "RemoveContainer" containerID="79a62cc84b57350d1b511fad54191076b7a7c36e2a58e09bd16e1d0264c18da1" Nov 02 08:56:01.662260 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:56:01.662240 5899 scope.go:122] "RemoveContainer" containerID="277ab3515aed2d276036ec6beff5a19d1a4c77ba06e594d6b60025e1c1cab480" Nov 02 08:56:02.029320 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:56:02.029280 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg"] Nov 02 08:56:02.042263 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:56:02.042229 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4dvdg"] Nov 02 08:56:03.818171 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:56:03.818135 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9bb891f-fa7a-4722-93ee-6785cd875939" path="/var/lib/kubelet/pods/a9bb891f-fa7a-4722-93ee-6785cd875939/volumes" Nov 02 08:56:12.630123 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:56:12.630084 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:56:13.140708 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:56:13.140667 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:56:20.027750 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:56:20.027714 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g"] Nov 02 08:56:20.046986 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:56:20.040978 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qbr7g"] Nov 02 08:56:21.818356 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:56:21.818319 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8126dd7-bce4-4574-afac-4b952c95a9c2" path="/var/lib/kubelet/pods/c8126dd7-bce4-4574-afac-4b952c95a9c2/volumes" Nov 02 08:56:33.030675 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:56:33.030636 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4"] Nov 02 08:56:33.043561 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:56:33.043529 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-clhf4"] Nov 02 08:56:33.817769 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:56:33.817729 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49ddd58-9d92-4fa3-b592-ca66ae1191d2" path="/var/lib/kubelet/pods/b49ddd58-9d92-4fa3-b592-ca66ae1191d2/volumes" Nov 02 08:56:43.141637 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:56:43.141554 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:57:01.703185 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:01.702830 5899 scope.go:122] "RemoveContainer" containerID="fe068a9c2f23bcb9cbc2a2c8a8fdbe134c83dcb5c208bac2b27b39fb0947b284" Nov 02 08:57:01.730049 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:01.730018 5899 scope.go:122] "RemoveContainer" containerID="6f3d0cf227eaca2c7b6c1de3f380678c58f09d2c48f005eb69cbac77bbff5c6a" Nov 02 08:57:01.741828 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:01.741801 5899 scope.go:122] "RemoveContainer" containerID="1134a057ef4af3451e57aaa87fe08ce55276caee2dcd6d9624ac276625a75f75" Nov 02 08:57:04.364076 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.364042 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8"] Nov 02 08:57:04.364502 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.364469 5899 cpu_manager.go:414] "containerMap: removing container" podUID="5aaa8786-c58b-461f-9e3d-baccc44b17a6" containerName="revoke-keys" Nov 02 08:57:04.364502 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.364482 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aaa8786-c58b-461f-9e3d-baccc44b17a6" containerName="revoke-keys" Nov 02 08:57:04.364584 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.364556 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aaa8786-c58b-461f-9e3d-baccc44b17a6" containerName="revoke-keys" Nov 02 08:57:04.369129 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.369105 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" Nov 02 08:57:04.400181 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.400131 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:57:04.400378 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.400205 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:57:04.401949 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.401916 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8"] Nov 02 08:57:04.427145 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.427105 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qkgk8\" (UID: \"5166fcad-b833-4815-9334-32cfd4c122bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" Nov 02 08:57:04.427145 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.427146 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qkgk8\" (UID: \"5166fcad-b833-4815-9334-32cfd4c122bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" Nov 02 08:57:04.427402 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.427321 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfwf5\" (UniqueName: \"kubernetes.io/projected/5166fcad-b833-4815-9334-32cfd4c122bd-kube-api-access-wfwf5\") pod \"maas-api-revoke-sub-d576e68b-qkgk8\" (UID: \"5166fcad-b833-4815-9334-32cfd4c122bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" Nov 02 08:57:04.527864 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.527825 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qkgk8\" (UID: \"5166fcad-b833-4815-9334-32cfd4c122bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" Nov 02 08:57:04.527864 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.527864 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qkgk8\" (UID: \"5166fcad-b833-4815-9334-32cfd4c122bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" Nov 02 08:57:04.528189 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.527915 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfwf5\" (UniqueName: \"kubernetes.io/projected/5166fcad-b833-4815-9334-32cfd4c122bd-kube-api-access-wfwf5\") pod \"maas-api-revoke-sub-d576e68b-qkgk8\" (UID: \"5166fcad-b833-4815-9334-32cfd4c122bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" Nov 02 08:57:04.528613 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.528588 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qkgk8\" (UID: \"5166fcad-b833-4815-9334-32cfd4c122bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" Nov 02 08:57:04.530291 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.530271 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qkgk8\" (UID: \"5166fcad-b833-4815-9334-32cfd4c122bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" Nov 02 08:57:04.553121 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.553088 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfwf5\" (UniqueName: \"kubernetes.io/projected/5166fcad-b833-4815-9334-32cfd4c122bd-kube-api-access-wfwf5\") pod \"maas-api-revoke-sub-d576e68b-qkgk8\" (UID: \"5166fcad-b833-4815-9334-32cfd4c122bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" Nov 02 08:57:04.682601 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.682505 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" Nov 02 08:57:04.982241 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:04.982211 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8"] Nov 02 08:57:04.985127 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:57:04.985099 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5166fcad_b833_4815_9334_32cfd4c122bd.slice/crio-1ea63459b51bc15a0cc95da197f7a1acc95a8a685fedfcb351879973dab2fd70 WatchSource:0}: Error finding container 1ea63459b51bc15a0cc95da197f7a1acc95a8a685fedfcb351879973dab2fd70: Status 404 returned error can't find the container with id 1ea63459b51bc15a0cc95da197f7a1acc95a8a685fedfcb351879973dab2fd70 Nov 02 08:57:05.141235 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:05.141187 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" event={"ID":"5166fcad-b833-4815-9334-32cfd4c122bd","Type":"ContainerStarted","Data":"1ea63459b51bc15a0cc95da197f7a1acc95a8a685fedfcb351879973dab2fd70"} Nov 02 08:57:06.148938 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:06.148853 5899 generic.go:356] "Generic (PLEG): container finished" podID="5166fcad-b833-4815-9334-32cfd4c122bd" containerID="39c459cf06f17b4c45d073f1cb482da15b4a1eea029879b728eb16d9aa09e1aa" exitCode=0 Nov 02 08:57:06.148938 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:06.148900 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" event={"ID":"5166fcad-b833-4815-9334-32cfd4c122bd","Type":"ContainerDied","Data":"39c459cf06f17b4c45d073f1cb482da15b4a1eea029879b728eb16d9aa09e1aa"} Nov 02 08:57:07.295734 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:07.295706 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" Nov 02 08:57:07.357117 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:07.357069 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-xks-ca\") pod \"5166fcad-b833-4815-9334-32cfd4c122bd\" (UID: \"5166fcad-b833-4815-9334-32cfd4c122bd\") " Nov 02 08:57:07.357336 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:07.357249 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-openshift-ca\") pod \"5166fcad-b833-4815-9334-32cfd4c122bd\" (UID: \"5166fcad-b833-4815-9334-32cfd4c122bd\") " Nov 02 08:57:07.357418 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:07.357339 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5166fcad-b833-4815-9334-32cfd4c122bd-kube-api-access-wfwf5\" (UniqueName: \"kubernetes.io/projected/5166fcad-b833-4815-9334-32cfd4c122bd-kube-api-access-wfwf5\") pod \"5166fcad-b833-4815-9334-32cfd4c122bd\" (UID: \"5166fcad-b833-4815-9334-32cfd4c122bd\") " Nov 02 08:57:07.357654 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:07.357623 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-openshift-ca" pod "5166fcad-b833-4815-9334-32cfd4c122bd" (UID: "5166fcad-b833-4815-9334-32cfd4c122bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:57:07.359248 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:07.359218 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-xks-ca" pod "5166fcad-b833-4815-9334-32cfd4c122bd" (UID: "5166fcad-b833-4815-9334-32cfd4c122bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:57:07.359355 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:07.359322 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5166fcad-b833-4815-9334-32cfd4c122bd-kube-api-access-wfwf5" pod "5166fcad-b833-4815-9334-32cfd4c122bd" (UID: "5166fcad-b833-4815-9334-32cfd4c122bd"). InnerVolumeSpecName "kube-api-access-wfwf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:57:07.458926 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:07.458840 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:57:07.458926 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:07.458873 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfwf5\" (UniqueName: \"kubernetes.io/projected/5166fcad-b833-4815-9334-32cfd4c122bd-kube-api-access-wfwf5\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:57:07.458926 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:07.458885 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5166fcad-b833-4815-9334-32cfd4c122bd-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:57:08.162149 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:08.162069 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" event={"ID":"5166fcad-b833-4815-9334-32cfd4c122bd","Type":"ContainerDied","Data":"1ea63459b51bc15a0cc95da197f7a1acc95a8a685fedfcb351879973dab2fd70"} Nov 02 08:57:08.162149 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:08.162107 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8" Nov 02 08:57:08.162149 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:08.162120 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ea63459b51bc15a0cc95da197f7a1acc95a8a685fedfcb351879973dab2fd70" Nov 02 08:57:12.672411 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:12.672355 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:57:13.140289 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:57:13.140254 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:57:19.280673 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.280615 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9"] Nov 02 08:57:19.281403 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.281377 5899 cpu_manager.go:414] "containerMap: removing container" podUID="5166fcad-b833-4815-9334-32cfd4c122bd" containerName="revoke-keys" Nov 02 08:57:19.281513 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.281404 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5166fcad-b833-4815-9334-32cfd4c122bd" containerName="revoke-keys" Nov 02 08:57:19.281584 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.281528 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="5166fcad-b833-4815-9334-32cfd4c122bd" containerName="revoke-keys" Nov 02 08:57:19.299418 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.299377 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" Nov 02 08:57:19.304802 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.304766 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9"] Nov 02 08:57:19.315683 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.315654 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 08:57:19.315844 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.315787 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\"" Nov 02 08:57:19.380257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.380210 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db56934d-9303-4779-9123-82cc3a712725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6xpv9\" (UID: \"db56934d-9303-4779-9123-82cc3a712725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" Nov 02 08:57:19.380468 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.380294 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db56934d-9303-4779-9123-82cc3a712725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6xpv9\" (UID: \"db56934d-9303-4779-9123-82cc3a712725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" Nov 02 08:57:19.380468 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.380329 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2ngd\" (UniqueName: \"kubernetes.io/projected/db56934d-9303-4779-9123-82cc3a712725-kube-api-access-m2ngd\") pod \"maas-api-revoke-sub-d2bc93bc-6xpv9\" (UID: \"db56934d-9303-4779-9123-82cc3a712725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" Nov 02 08:57:19.481994 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.481943 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db56934d-9303-4779-9123-82cc3a712725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6xpv9\" (UID: \"db56934d-9303-4779-9123-82cc3a712725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" Nov 02 08:57:19.482204 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.482026 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db56934d-9303-4779-9123-82cc3a712725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6xpv9\" (UID: \"db56934d-9303-4779-9123-82cc3a712725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" Nov 02 08:57:19.482204 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.482063 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2ngd\" (UniqueName: \"kubernetes.io/projected/db56934d-9303-4779-9123-82cc3a712725-kube-api-access-m2ngd\") pod \"maas-api-revoke-sub-d2bc93bc-6xpv9\" (UID: \"db56934d-9303-4779-9123-82cc3a712725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" Nov 02 08:57:19.482907 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.482878 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db56934d-9303-4779-9123-82cc3a712725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6xpv9\" (UID: \"db56934d-9303-4779-9123-82cc3a712725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" Nov 02 08:57:19.484656 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.484625 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db56934d-9303-4779-9123-82cc3a712725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6xpv9\" (UID: \"db56934d-9303-4779-9123-82cc3a712725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" Nov 02 08:57:19.517275 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.517188 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2ngd\" (UniqueName: \"kubernetes.io/projected/db56934d-9303-4779-9123-82cc3a712725-kube-api-access-m2ngd\") pod \"maas-api-revoke-sub-d2bc93bc-6xpv9\" (UID: \"db56934d-9303-4779-9123-82cc3a712725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" Nov 02 08:57:19.626693 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:19.626335 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" Nov 02 08:57:20.033278 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:20.022185 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9"] Nov 02 08:57:20.036611 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:57:20.036573 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb56934d_9303_4779_9123_82cc3a712725.slice/crio-933e9e5054b75589958ceb6e78040c36a27d1c7f61234eacf6513bf1786c6858 WatchSource:0}: Error finding container 933e9e5054b75589958ceb6e78040c36a27d1c7f61234eacf6513bf1786c6858: Status 404 returned error can't find the container with id 933e9e5054b75589958ceb6e78040c36a27d1c7f61234eacf6513bf1786c6858 Nov 02 08:57:20.231708 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:20.231668 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" event={"ID":"db56934d-9303-4779-9123-82cc3a712725","Type":"ContainerStarted","Data":"933e9e5054b75589958ceb6e78040c36a27d1c7f61234eacf6513bf1786c6858"} Nov 02 08:57:21.239705 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:21.239662 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" event={"ID":"db56934d-9303-4779-9123-82cc3a712725","Type":"ContainerStarted","Data":"00bfe2499cc66c1614a4a930318e97fb6d344f249b610e2572fdeffa5b71f2de"} Nov 02 08:57:21.283458 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:21.283411 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" podStartSLOduration=2.283396245 podStartE2EDuration="2.283396245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:57:21.281922926 +0000 UTC m=+5541.580203744" watchObservedRunningTime="2026-11-02 08:57:21.283396245 +0000 UTC m=+5541.581677069" Nov 02 08:57:22.246623 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:22.246586 5899 generic.go:356] "Generic (PLEG): container finished" podID="db56934d-9303-4779-9123-82cc3a712725" containerID="00bfe2499cc66c1614a4a930318e97fb6d344f249b610e2572fdeffa5b71f2de" exitCode=0 Nov 02 08:57:22.247038 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:22.246669 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" event={"ID":"db56934d-9303-4779-9123-82cc3a712725","Type":"ContainerDied","Data":"00bfe2499cc66c1614a4a930318e97fb6d344f249b610e2572fdeffa5b71f2de"} Nov 02 08:57:23.386808 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:23.386769 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" Nov 02 08:57:23.483081 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:23.483044 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db56934d-9303-4779-9123-82cc3a712725-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db56934d-9303-4779-9123-82cc3a712725-maas-api-xks-ca\") pod \"db56934d-9303-4779-9123-82cc3a712725\" (UID: \"db56934d-9303-4779-9123-82cc3a712725\") " Nov 02 08:57:23.483260 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:23.483225 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db56934d-9303-4779-9123-82cc3a712725-kube-api-access-m2ngd\" (UniqueName: \"kubernetes.io/projected/db56934d-9303-4779-9123-82cc3a712725-kube-api-access-m2ngd\") pod \"db56934d-9303-4779-9123-82cc3a712725\" (UID: \"db56934d-9303-4779-9123-82cc3a712725\") " Nov 02 08:57:23.483349 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:23.483295 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db56934d-9303-4779-9123-82cc3a712725-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db56934d-9303-4779-9123-82cc3a712725-maas-api-openshift-ca\") pod \"db56934d-9303-4779-9123-82cc3a712725\" (UID: \"db56934d-9303-4779-9123-82cc3a712725\") " Nov 02 08:57:23.483712 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:23.483687 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db56934d-9303-4779-9123-82cc3a712725-maas-api-openshift-ca" pod "db56934d-9303-4779-9123-82cc3a712725" (UID: "db56934d-9303-4779-9123-82cc3a712725"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:57:23.485139 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:23.485115 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db56934d-9303-4779-9123-82cc3a712725-maas-api-xks-ca" pod "db56934d-9303-4779-9123-82cc3a712725" (UID: "db56934d-9303-4779-9123-82cc3a712725"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:57:23.485253 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:23.485233 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db56934d-9303-4779-9123-82cc3a712725-kube-api-access-m2ngd" pod "db56934d-9303-4779-9123-82cc3a712725" (UID: "db56934d-9303-4779-9123-82cc3a712725"). InnerVolumeSpecName "kube-api-access-m2ngd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:57:23.584501 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:23.584406 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2ngd\" (UniqueName: \"kubernetes.io/projected/db56934d-9303-4779-9123-82cc3a712725-kube-api-access-m2ngd\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:57:23.584501 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:23.584441 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db56934d-9303-4779-9123-82cc3a712725-maas-api-openshift-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:57:23.584501 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:23.584455 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db56934d-9303-4779-9123-82cc3a712725-maas-api-xks-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:57:24.257824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:24.257776 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" event={"ID":"db56934d-9303-4779-9123-82cc3a712725","Type":"ContainerDied","Data":"933e9e5054b75589958ceb6e78040c36a27d1c7f61234eacf6513bf1786c6858"} Nov 02 08:57:24.257824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:24.257827 5899 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="933e9e5054b75589958ceb6e78040c36a27d1c7f61234eacf6513bf1786c6858" Nov 02 08:57:24.257824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:24.257789 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9" Nov 02 08:57:31.394070 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.394035 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-699d9fc559-82ftd"] Nov 02 08:57:31.394529 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.394513 5899 cpu_manager.go:414] "containerMap: removing container" podUID="db56934d-9303-4779-9123-82cc3a712725" containerName="revoke-keys" Nov 02 08:57:31.394575 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.394530 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db56934d-9303-4779-9123-82cc3a712725" containerName="revoke-keys" Nov 02 08:57:31.394622 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.394612 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="db56934d-9303-4779-9123-82cc3a712725" containerName="revoke-keys" Nov 02 08:57:31.398013 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.397994 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:31.422640 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.422597 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4tgq8\"" Nov 02 08:57:31.435954 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.435922 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 08:57:31.436133 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.436069 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 08:57:31.441669 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.441636 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-699d9fc559-82ftd"] Nov 02 08:57:31.562332 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.562292 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-metrics-certs\") pod \"maas-api-699d9fc559-82ftd\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:31.562495 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.562345 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmsdr\" (UniqueName: \"kubernetes.io/projected/969fc463-429b-478c-abeb-992e3618a20e-kube-api-access-dmsdr\") pod \"maas-api-699d9fc559-82ftd\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:31.562495 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.562403 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-maas-api-tls\") pod \"maas-api-699d9fc559-82ftd\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:31.562591 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.562533 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/969fc463-429b-478c-abeb-992e3618a20e-openshift-service-ca\") pod \"maas-api-699d9fc559-82ftd\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:31.663726 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.663618 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/969fc463-429b-478c-abeb-992e3618a20e-openshift-service-ca\") pod \"maas-api-699d9fc559-82ftd\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:31.663726 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.663711 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-metrics-certs\") pod \"maas-api-699d9fc559-82ftd\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:31.664026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.663750 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmsdr\" (UniqueName: \"kubernetes.io/projected/969fc463-429b-478c-abeb-992e3618a20e-kube-api-access-dmsdr\") pod \"maas-api-699d9fc559-82ftd\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:31.664026 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.663774 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-maas-api-tls\") pod \"maas-api-699d9fc559-82ftd\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:31.664461 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.664431 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/969fc463-429b-478c-abeb-992e3618a20e-openshift-service-ca\") pod \"maas-api-699d9fc559-82ftd\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:31.666293 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.666269 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-maas-api-tls\") pod \"maas-api-699d9fc559-82ftd\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:31.666365 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.666352 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-metrics-certs\") pod \"maas-api-699d9fc559-82ftd\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:31.689895 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.689867 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmsdr\" (UniqueName: \"kubernetes.io/projected/969fc463-429b-478c-abeb-992e3618a20e-kube-api-access-dmsdr\") pod \"maas-api-699d9fc559-82ftd\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:31.712440 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:31.712407 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:32.015477 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:32.015448 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-699d9fc559-82ftd"] Nov 02 08:57:32.017798 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:57:32.017771 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969fc463_429b_478c_abeb_992e3618a20e.slice/crio-357334155c7248be58e8148ac38413599ca482530629200e07817492b2ef73a5 WatchSource:0}: Error finding container 357334155c7248be58e8148ac38413599ca482530629200e07817492b2ef73a5: Status 404 returned error can't find the container with id 357334155c7248be58e8148ac38413599ca482530629200e07817492b2ef73a5 Nov 02 08:57:32.019021 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:32.019002 5899 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 08:57:32.045782 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:32.045744 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7"] Nov 02 08:57:32.055088 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:32.055054 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-22zz7"] Nov 02 08:57:32.301453 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:32.301364 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" event={"ID":"969fc463-429b-478c-abeb-992e3618a20e","Type":"ContainerStarted","Data":"357334155c7248be58e8148ac38413599ca482530629200e07817492b2ef73a5"} Nov 02 08:57:33.309527 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:33.309488 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" event={"ID":"969fc463-429b-478c-abeb-992e3618a20e","Type":"ContainerStarted","Data":"8e74c5ec14cb53642e9ae05279172798613f479ea2a5f15c078538692257515e"} Nov 02 08:57:33.310075 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:33.309581 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:33.355037 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:33.354982 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" podStartSLOduration=1.870133657 podStartE2EDuration="2.354967175s" totalImagesPullingTime="484.833518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:57:33.353206938 +0000 UTC m=+5553.651487764" watchObservedRunningTime="2026-11-02 08:57:33.354967175 +0000 UTC m=+5553.653248001" Nov 02 08:57:33.818409 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:33.818372 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da64d6d-8df4-451f-a617-1fa6ad8379a4" path="/var/lib/kubelet/pods/9da64d6d-8df4-451f-a617-1fa6ad8379a4/volumes" Nov 02 08:57:38.743060 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.743019 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-699d9fc559-82ftd"] Nov 02 08:57:38.743656 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.743281 5899 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" podUID="969fc463-429b-478c-abeb-992e3618a20e" containerName="maas-api" containerID="cri-o://8e74c5ec14cb53642e9ae05279172798613f479ea2a5f15c078538692257515e" gracePeriod=30 Nov 02 08:57:38.749735 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.749710 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:38.782211 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.782176 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx"] Nov 02 08:57:38.785806 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.785781 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:38.809280 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.809237 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx"] Nov 02 08:57:38.830280 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.830244 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br9mq\" (UniqueName: \"kubernetes.io/projected/cc41b496-4db5-47e9-aac0-500aefe1fc91-kube-api-access-br9mq\") pod \"maas-api-75cb49fd59-5w7zx\" (UID: \"cc41b496-4db5-47e9-aac0-500aefe1fc91\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:38.830454 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.830290 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc41b496-4db5-47e9-aac0-500aefe1fc91-maas-api-tls\") pod \"maas-api-75cb49fd59-5w7zx\" (UID: \"cc41b496-4db5-47e9-aac0-500aefe1fc91\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:38.830454 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.830311 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc41b496-4db5-47e9-aac0-500aefe1fc91-openshift-service-ca\") pod \"maas-api-75cb49fd59-5w7zx\" (UID: \"cc41b496-4db5-47e9-aac0-500aefe1fc91\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:38.830454 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.830417 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc41b496-4db5-47e9-aac0-500aefe1fc91-metrics-certs\") pod \"maas-api-75cb49fd59-5w7zx\" (UID: \"cc41b496-4db5-47e9-aac0-500aefe1fc91\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:38.931012 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.930959 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc41b496-4db5-47e9-aac0-500aefe1fc91-openshift-service-ca\") pod \"maas-api-75cb49fd59-5w7zx\" (UID: \"cc41b496-4db5-47e9-aac0-500aefe1fc91\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:38.931199 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.931063 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc41b496-4db5-47e9-aac0-500aefe1fc91-metrics-certs\") pod \"maas-api-75cb49fd59-5w7zx\" (UID: \"cc41b496-4db5-47e9-aac0-500aefe1fc91\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:38.931257 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.931221 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9mq\" (UniqueName: \"kubernetes.io/projected/cc41b496-4db5-47e9-aac0-500aefe1fc91-kube-api-access-br9mq\") pod \"maas-api-75cb49fd59-5w7zx\" (UID: \"cc41b496-4db5-47e9-aac0-500aefe1fc91\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:38.931302 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.931260 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc41b496-4db5-47e9-aac0-500aefe1fc91-maas-api-tls\") pod \"maas-api-75cb49fd59-5w7zx\" (UID: \"cc41b496-4db5-47e9-aac0-500aefe1fc91\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:38.931864 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.931836 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc41b496-4db5-47e9-aac0-500aefe1fc91-openshift-service-ca\") pod \"maas-api-75cb49fd59-5w7zx\" (UID: \"cc41b496-4db5-47e9-aac0-500aefe1fc91\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:38.933638 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.933603 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc41b496-4db5-47e9-aac0-500aefe1fc91-metrics-certs\") pod \"maas-api-75cb49fd59-5w7zx\" (UID: \"cc41b496-4db5-47e9-aac0-500aefe1fc91\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:38.933796 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.933779 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc41b496-4db5-47e9-aac0-500aefe1fc91-maas-api-tls\") pod \"maas-api-75cb49fd59-5w7zx\" (UID: \"cc41b496-4db5-47e9-aac0-500aefe1fc91\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:38.956099 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.956068 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br9mq\" (UniqueName: \"kubernetes.io/projected/cc41b496-4db5-47e9-aac0-500aefe1fc91-kube-api-access-br9mq\") pod \"maas-api-75cb49fd59-5w7zx\" (UID: \"cc41b496-4db5-47e9-aac0-500aefe1fc91\") " pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:38.991594 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:38.991570 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:39.032621 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.032539 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-maas-api-tls\") pod \"969fc463-429b-478c-abeb-992e3618a20e\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " Nov 02 08:57:39.032621 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.032599 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/969fc463-429b-478c-abeb-992e3618a20e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/969fc463-429b-478c-abeb-992e3618a20e-openshift-service-ca\") pod \"969fc463-429b-478c-abeb-992e3618a20e\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " Nov 02 08:57:39.032824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.032625 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-metrics-certs\") pod \"969fc463-429b-478c-abeb-992e3618a20e\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " Nov 02 08:57:39.032824 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.032718 5899 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969fc463-429b-478c-abeb-992e3618a20e-kube-api-access-dmsdr\" (UniqueName: \"kubernetes.io/projected/969fc463-429b-478c-abeb-992e3618a20e-kube-api-access-dmsdr\") pod \"969fc463-429b-478c-abeb-992e3618a20e\" (UID: \"969fc463-429b-478c-abeb-992e3618a20e\") " Nov 02 08:57:39.033055 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.033027 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969fc463-429b-478c-abeb-992e3618a20e-openshift-service-ca" pod "969fc463-429b-478c-abeb-992e3618a20e" (UID: "969fc463-429b-478c-abeb-992e3618a20e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 08:57:39.034748 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.034719 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-maas-api-tls" pod "969fc463-429b-478c-abeb-992e3618a20e" (UID: "969fc463-429b-478c-abeb-992e3618a20e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:57:39.034871 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.034762 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-metrics-certs" pod "969fc463-429b-478c-abeb-992e3618a20e" (UID: "969fc463-429b-478c-abeb-992e3618a20e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 08:57:39.034871 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.034778 5899 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969fc463-429b-478c-abeb-992e3618a20e-kube-api-access-dmsdr" pod "969fc463-429b-478c-abeb-992e3618a20e" (UID: "969fc463-429b-478c-abeb-992e3618a20e"). InnerVolumeSpecName "kube-api-access-dmsdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 08:57:39.134387 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.134353 5899 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmsdr\" (UniqueName: \"kubernetes.io/projected/969fc463-429b-478c-abeb-992e3618a20e-kube-api-access-dmsdr\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:57:39.134387 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.134382 5899 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-maas-api-tls\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:57:39.134387 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.134394 5899 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/969fc463-429b-478c-abeb-992e3618a20e-openshift-service-ca\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:57:39.134639 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.134403 5899 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/969fc463-429b-478c-abeb-992e3618a20e-metrics-certs\") on node \"ip-10-0-135-6.ec2.internal\" DevicePath \"\"" Nov 02 08:57:39.148385 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.148342 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:39.341724 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.341637 5899 generic.go:356] "Generic (PLEG): container finished" podID="969fc463-429b-478c-abeb-992e3618a20e" containerID="8e74c5ec14cb53642e9ae05279172798613f479ea2a5f15c078538692257515e" exitCode=0 Nov 02 08:57:39.341724 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.341682 5899 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" Nov 02 08:57:39.341940 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.341717 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" event={"ID":"969fc463-429b-478c-abeb-992e3618a20e","Type":"ContainerDied","Data":"8e74c5ec14cb53642e9ae05279172798613f479ea2a5f15c078538692257515e"} Nov 02 08:57:39.341940 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.341769 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699d9fc559-82ftd" event={"ID":"969fc463-429b-478c-abeb-992e3618a20e","Type":"ContainerDied","Data":"357334155c7248be58e8148ac38413599ca482530629200e07817492b2ef73a5"} Nov 02 08:57:39.341940 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.341790 5899 scope.go:122] "RemoveContainer" containerID="8e74c5ec14cb53642e9ae05279172798613f479ea2a5f15c078538692257515e" Nov 02 08:57:39.351058 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.351038 5899 scope.go:122] "RemoveContainer" containerID="8e74c5ec14cb53642e9ae05279172798613f479ea2a5f15c078538692257515e" Nov 02 08:57:39.351348 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:57:39.351329 5899 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e74c5ec14cb53642e9ae05279172798613f479ea2a5f15c078538692257515e\": container with ID starting with 8e74c5ec14cb53642e9ae05279172798613f479ea2a5f15c078538692257515e not found: ID does not exist" containerID="8e74c5ec14cb53642e9ae05279172798613f479ea2a5f15c078538692257515e" Nov 02 08:57:39.351409 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.351359 5899 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e74c5ec14cb53642e9ae05279172798613f479ea2a5f15c078538692257515e"} err="failed to get container status \"8e74c5ec14cb53642e9ae05279172798613f479ea2a5f15c078538692257515e\": rpc error: code = NotFound desc = could not find container \"8e74c5ec14cb53642e9ae05279172798613f479ea2a5f15c078538692257515e\": container with ID starting with 8e74c5ec14cb53642e9ae05279172798613f479ea2a5f15c078538692257515e not found: ID does not exist" Nov 02 08:57:39.400392 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.400363 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-699d9fc559-82ftd"] Nov 02 08:57:39.412644 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.412615 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-699d9fc559-82ftd"] Nov 02 08:57:39.448016 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.447886 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx"] Nov 02 08:57:39.450853 ip-10-0-135-6 kubenswrapper[5899]: W1102 08:57:39.450826 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc41b496_4db5_47e9_aac0_500aefe1fc91.slice/crio-f2bc914c1cc5e4e96006937a95fc1dee017d69cbec508bf921a79f62bb719d3b WatchSource:0}: Error finding container f2bc914c1cc5e4e96006937a95fc1dee017d69cbec508bf921a79f62bb719d3b: Status 404 returned error can't find the container with id f2bc914c1cc5e4e96006937a95fc1dee017d69cbec508bf921a79f62bb719d3b Nov 02 08:57:39.819001 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:39.818964 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="969fc463-429b-478c-abeb-992e3618a20e" path="/var/lib/kubelet/pods/969fc463-429b-478c-abeb-992e3618a20e/volumes" Nov 02 08:57:40.350382 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:40.350345 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" event={"ID":"cc41b496-4db5-47e9-aac0-500aefe1fc91","Type":"ContainerStarted","Data":"3d84d1b6582fafe6d41378d099af60620ff0100cda5f49443d91741a53851b7e"} Nov 02 08:57:40.350592 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:40.350401 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" event={"ID":"cc41b496-4db5-47e9-aac0-500aefe1fc91","Type":"ContainerStarted","Data":"f2bc914c1cc5e4e96006937a95fc1dee017d69cbec508bf921a79f62bb719d3b"} Nov 02 08:57:40.350592 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:40.350472 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:40.395457 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:40.395401 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" podStartSLOduration=2.001217794 podStartE2EDuration="2.395386569s" totalImagesPullingTime="394.168775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:57:40.393655834 +0000 UTC m=+5560.691936659" watchObservedRunningTime="2026-11-02 08:57:40.395386569 +0000 UTC m=+5560.693667393" Nov 02 08:57:43.050002 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:43.049958 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz"] Nov 02 08:57:43.061304 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:43.061261 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2c4kz"] Nov 02 08:57:43.152797 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:57:43.152743 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:57:43.831655 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:43.831617 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71dcd9a8-5475-4d16-a972-eec434a90f66" path="/var/lib/kubelet/pods/71dcd9a8-5475-4d16-a972-eec434a90f66/volumes" Nov 02 08:57:46.361891 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:46.361860 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-75cb49fd59-5w7zx" Nov 02 08:57:50.957143 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:50.957107 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:57:51.097090 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:57:51.097052 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:58:01.837366 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:58:01.837330 5899 scope.go:122] "RemoveContainer" containerID="460d0863bf9a4d314c31e35fffa3b588682ad05d68715b6d07e1faf351ecf62b" Nov 02 08:58:01.846071 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:58:01.846050 5899 scope.go:122] "RemoveContainer" containerID="3d5e1f925a33a99f54368c1a222b85fee95cb67c20740396715c03e886d930ad" Nov 02 08:58:01.854243 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:58:01.854219 5899 scope.go:122] "RemoveContainer" containerID="17faebfd239fb470412b639426b6e528be7620b577406fd453038b7120af9cc6" Nov 02 08:58:01.861912 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:58:01.861894 5899 scope.go:122] "RemoveContainer" containerID="37855e38c3d1126333e24c9c1a893bbd53d2dc289143722c9bc8addc23eefd42" Nov 02 08:58:07.882747 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:58:07.882707 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:58:08.021310 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:58:08.021278 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:58:12.998115 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:58:12.998035 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:58:13.140504 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:58:13.140469 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:58:43.151796 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:58:43.151749 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:59:01.897561 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:59:01.897526 5899 scope.go:122] "RemoveContainer" containerID="7b72dafd43786a0620ef0d405e712958060c97f1098a2f18913c0212c892f5b6" Nov 02 08:59:01.906535 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:59:01.906515 5899 scope.go:122] "RemoveContainer" containerID="7b57ad3ef5d38fc731756cbd4ea905236d2c8b7cbba08fd19176b4e2ed945ed3" Nov 02 08:59:01.914588 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:59:01.914570 5899 scope.go:122] "RemoveContainer" containerID="85c7b74815b6453ef1ef56daaa5f0e8fcaa2befe5ca6d23f2dafceaf96750892" Nov 02 08:59:08.033036 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:59:08.032997 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q"] Nov 02 08:59:08.045728 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:59:08.045699 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d0f0d4-s7h2q"] Nov 02 08:59:09.818704 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:59:09.818670 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99094ba7-68dd-4e7c-b1a7-61332f95051d" path="/var/lib/kubelet/pods/99094ba7-68dd-4e7c-b1a7-61332f95051d/volumes" Nov 02 08:59:12.934580 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:59:12.934513 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 08:59:13.140598 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:59:13.140563 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:59:43.140520 ip-10-0-135-6 kubenswrapper[5899]: E1102 08:59:43.140441 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 08:59:58.964198 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:59:58.963345 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7xqxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 08:59:58.964198 ip-10-0-135-6 kubenswrapper[5899]: I1102 08:59:58.963414 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" podUID="e43d749b-223f-4a0a-997a-a0637ddf2d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:00:01.375973 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:00:01.375935 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:00:01.513542 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:00:01.513506 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:00:08.777955 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:00:08.777895 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:00:08.948778 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:00:08.948742 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:00:10.279016 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:00:10.278975 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:00:13.048009 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:00:13.047969 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:00:13.139892 ip-10-0-135-6 kubenswrapper[5899]: E1102 09:00:13.139851 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 09:00:13.183048 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:00:13.183001 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:00:23.035988 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:00:23.035929 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6"] Nov 02 09:00:23.042846 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:00:23.042809 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wksq6"] Nov 02 09:00:23.830957 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:00:23.830522 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aaa8786-c58b-461f-9e3d-baccc44b17a6" path="/var/lib/kubelet/pods/5aaa8786-c58b-461f-9e3d-baccc44b17a6/volumes" Nov 02 09:00:43.140938 ip-10-0-135-6 kubenswrapper[5899]: E1102 09:00:43.140896 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 09:00:48.407733 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:00:48.407696 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:01:01.952000 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:01:01.951959 5899 scope.go:122] "RemoveContainer" containerID="fa801c97a1d7f37aafbd2c3217498ffc3d7ab300be71a53a2da8b927b4e49204" Nov 02 09:01:06.746344 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:01:06.746298 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cf7zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:01:06.746858 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:01:06.746382 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cf7zm" podUID="b0c571f9-9dc1-493e-b1e5-a256cad13493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:01:13.140693 ip-10-0-135-6 kubenswrapper[5899]: E1102 09:01:13.140611 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 09:01:13.178111 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:01:13.178073 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:01:43.140995 ip-10-0-135-6 kubenswrapper[5899]: E1102 09:01:43.140956 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 09:02:01.973246 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:02:01.973212 5899 scope.go:122] "RemoveContainer" containerID="13329a38d2c0a5def6a1ab10939de15b93e402d4434b202429b47b4036eb72b5" Nov 02 09:02:01.982242 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:02:01.982218 5899 scope.go:122] "RemoveContainer" containerID="b504738262590d625877f11ba7f0fc57f31fb389f213013860f52bff3c2042cf" Nov 02 09:02:08.039585 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:02:08.039541 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8"] Nov 02 09:02:08.051849 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:02:08.051812 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qkgk8"] Nov 02 09:02:09.818603 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:02:09.818570 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5166fcad-b833-4815-9334-32cfd4c122bd" path="/var/lib/kubelet/pods/5166fcad-b833-4815-9334-32cfd4c122bd/volumes" Nov 02 09:02:13.142131 ip-10-0-135-6 kubenswrapper[5899]: E1102 09:02:13.142091 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 09:02:13.329584 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:02:13.329541 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:02:24.028492 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:02:24.028451 5899 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9"] Nov 02 09:02:24.041252 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:02:24.041218 5899 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6xpv9"] Nov 02 09:02:25.818669 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:02:25.818630 5899 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db56934d-9303-4779-9123-82cc3a712725" path="/var/lib/kubelet/pods/db56934d-9303-4779-9123-82cc3a712725/volumes" Nov 02 09:02:28.963186 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:02:28.959274 5899 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-7xqxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:02:28.963186 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:02:28.959366 5899 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-7xqxx" podUID="e43d749b-223f-4a0a-997a-a0637ddf2d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:02:43.141039 ip-10-0-135-6 kubenswrapper[5899]: E1102 09:02:43.140955 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 09:03:00.597652 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:00.597616 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-g8nbx_5ac39273-9863-4a60-86f4-29e64cbe2946/manager/0.log" Nov 02 09:03:00.789541 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:00.789494 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kv7wp_62de7c44-3318-4553-b3d0-f0124ed28ac4/manager/0.log" Nov 02 09:03:00.993632 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:00.993597 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-kqv85_580bc81e-1b83-472a-9af3-153ba348eab7/manager/0.log" Nov 02 09:03:01.497623 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:01.497588 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pq62z_787dc60b-c7be-4b20-9b11-daf3140a7589/server/0.log" Nov 02 09:03:02.834689 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:02.834645 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-75cb49fd59-5w7zx_cc41b496-4db5-47e9-aac0-500aefe1fc91/maas-api/0.log" Nov 02 09:03:06.566729 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:06.566688 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-mt86t_1a348850-1b82-4852-9cf3-be29bbb75e6e/postgres/0.log" Nov 02 09:03:09.038154 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:09.038122 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xjq52_b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae/kuadrant-console-plugin/0.log" Nov 02 09:03:11.040525 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:11.040490 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6946f9c7-gkrcg_ea129fbd-3cf5-4889-a684-4c55c0c13c1b/kube-auth-proxy/0.log" Nov 02 09:03:11.592436 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:11.592405 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-7xqxx_e43d749b-223f-4a0a-997a-a0637ddf2d00/istio-proxy/0.log" Nov 02 09:03:11.788710 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:11.788673 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-cf7zm_b0c571f9-9dc1-493e-b1e5-a256cad13493/istio-proxy/0.log" Nov 02 09:03:11.972375 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:11.972292 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-hkc5n_14f48878-440e-45ac-900a-a29238f348c9/istio-proxy/0.log" Nov 02 09:03:13.140390 ip-10-0-135-6 kubenswrapper[5899]: E1102 09:03:13.140349 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 09:03:13.481774 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:13.481734 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:03:26.355963 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:26.355916 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:03:26.510083 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:26.510046 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:03:30.768069 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:30.768035 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9ztwm_253a4509-48e5-48c2-8c13-2dc361e5763f/global-pull-secret-syncer/0.log" Nov 02 09:03:31.702259 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:31.702224 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" Nov 02 09:03:32.222341 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:32.222308 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-6.ec2.internal_cb0aa2bb77fd4bd0612d1dffa702ab46/haproxy/0.log" Nov 02 09:03:40.626758 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:40.626729 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xjq52_b0407d3f-b9ba-4445-97ce-cb4ceb2e77ae/kuadrant-console-plugin/0.log" Nov 02 09:03:42.745986 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.745946 5899 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8"] Nov 02 09:03:42.746448 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.746432 5899 cpu_manager.go:414] "containerMap: removing container" podUID="969fc463-429b-478c-abeb-992e3618a20e" containerName="maas-api" Nov 02 09:03:42.746494 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.746449 5899 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969fc463-429b-478c-abeb-992e3618a20e" containerName="maas-api" Nov 02 09:03:42.746534 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.746527 5899 memory_manager.go:365] "RemoveStaleState removing state" podUID="969fc463-429b-478c-abeb-992e3618a20e" containerName="maas-api" Nov 02 09:03:42.749836 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.749814 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.771140 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.771110 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8"] Nov 02 09:03:42.782595 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.782570 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-25t77\"/\"kube-root-ca.crt\"" Nov 02 09:03:42.782595 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.782592 5899 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-25t77\"/\"default-dockercfg-tsbd8\"" Nov 02 09:03:42.782770 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.782574 5899 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-25t77\"/\"openshift-service-ca.crt\"" Nov 02 09:03:42.791213 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.791182 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/980f0fa7-409d-4e4d-ae57-9d38a9edf146-proc\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.791329 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.791255 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmrnc\" (UniqueName: \"kubernetes.io/projected/980f0fa7-409d-4e4d-ae57-9d38a9edf146-kube-api-access-dmrnc\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.791329 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.791290 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/980f0fa7-409d-4e4d-ae57-9d38a9edf146-podres\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.791329 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.791310 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/980f0fa7-409d-4e4d-ae57-9d38a9edf146-sys\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.791473 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.791442 5899 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/980f0fa7-409d-4e4d-ae57-9d38a9edf146-lib-modules\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.816424 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.816386 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b36155ab-82c8-456c-8a49-8baaf7006844/alertmanager/0.log" Nov 02 09:03:42.892898 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.892858 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrnc\" (UniqueName: \"kubernetes.io/projected/980f0fa7-409d-4e4d-ae57-9d38a9edf146-kube-api-access-dmrnc\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.893077 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.892904 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/980f0fa7-409d-4e4d-ae57-9d38a9edf146-podres\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.893077 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.892985 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/980f0fa7-409d-4e4d-ae57-9d38a9edf146-podres\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.893077 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.893019 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/980f0fa7-409d-4e4d-ae57-9d38a9edf146-sys\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.893227 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.893105 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/980f0fa7-409d-4e4d-ae57-9d38a9edf146-lib-modules\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.893227 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.893136 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/980f0fa7-409d-4e4d-ae57-9d38a9edf146-sys\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.893301 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.893245 5899 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/980f0fa7-409d-4e4d-ae57-9d38a9edf146-proc\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.893301 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.893247 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/980f0fa7-409d-4e4d-ae57-9d38a9edf146-lib-modules\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.893301 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.893274 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/980f0fa7-409d-4e4d-ae57-9d38a9edf146-proc\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.917546 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.917509 5899 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrnc\" (UniqueName: \"kubernetes.io/projected/980f0fa7-409d-4e4d-ae57-9d38a9edf146-kube-api-access-dmrnc\") pod \"perf-node-gather-daemonset-98lt8\" (UID: \"980f0fa7-409d-4e4d-ae57-9d38a9edf146\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:42.935768 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:42.935739 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b36155ab-82c8-456c-8a49-8baaf7006844/config-reloader/0.log" Nov 02 09:03:43.051924 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:43.051836 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b36155ab-82c8-456c-8a49-8baaf7006844/kube-rbac-proxy-web/0.log" Nov 02 09:03:43.062881 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:43.062854 5899 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:43.146544 ip-10-0-135-6 kubenswrapper[5899]: E1102 09:03:43.146480 5899 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b486r" containerName="extractor" Nov 02 09:03:43.174104 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:43.174075 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b36155ab-82c8-456c-8a49-8baaf7006844/kube-rbac-proxy/0.log" Nov 02 09:03:43.293535 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:43.293500 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b36155ab-82c8-456c-8a49-8baaf7006844/kube-rbac-proxy-metric/0.log" Nov 02 09:03:43.366064 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:43.365993 5899 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8"] Nov 02 09:03:43.369269 ip-10-0-135-6 kubenswrapper[5899]: W1102 09:03:43.369238 5899 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod980f0fa7_409d_4e4d_ae57_9d38a9edf146.slice/crio-2d13e2a3410fff07d840242188f67bff147d87a99972ec7ae070bdea94895d9c WatchSource:0}: Error finding container 2d13e2a3410fff07d840242188f67bff147d87a99972ec7ae070bdea94895d9c: Status 404 returned error can't find the container with id 2d13e2a3410fff07d840242188f67bff147d87a99972ec7ae070bdea94895d9c Nov 02 09:03:43.426389 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:43.426355 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" event={"ID":"980f0fa7-409d-4e4d-ae57-9d38a9edf146","Type":"ContainerStarted","Data":"2d13e2a3410fff07d840242188f67bff147d87a99972ec7ae070bdea94895d9c"} Nov 02 09:03:43.427181 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:43.427146 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b36155ab-82c8-456c-8a49-8baaf7006844/prom-label-proxy/0.log" Nov 02 09:03:43.564440 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:43.564412 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b36155ab-82c8-456c-8a49-8baaf7006844/init-config-reloader/0.log" Nov 02 09:03:43.817556 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:43.817519 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-sbrnh_0768e3b2-d790-4375-acf5-111424b1c5c3/kube-state-metrics/0.log" Nov 02 09:03:43.932013 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:43.931984 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-sbrnh_0768e3b2-d790-4375-acf5-111424b1c5c3/kube-rbac-proxy-main/0.log" Nov 02 09:03:44.050731 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:44.050689 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-sbrnh_0768e3b2-d790-4375-acf5-111424b1c5c3/kube-rbac-proxy-self/0.log" Nov 02 09:03:44.169111 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:44.169024 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-895bcdc89-gxp7k_0e577f03-1ea2-4496-9adb-84a2dc4dc72f/metrics-server/0.log" Nov 02 09:03:44.294554 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:44.294523 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-xfn8l_b9aa31c8-4e1b-4fb5-b26a-ab0089e364c0/monitoring-plugin/0.log" Nov 02 09:03:44.432969 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:44.432877 5899 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" event={"ID":"980f0fa7-409d-4e4d-ae57-9d38a9edf146","Type":"ContainerStarted","Data":"6b4a813382dc55c1c867d85fb7f770dd93eaf2d10cd0e4c79b49d7577e75e8c0"} Nov 02 09:03:44.432969 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:44.432942 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:44.513586 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:44.513515 5899 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" podStartSLOduration=2.5135015579999997 podStartE2EDuration="2.513501558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:03:44.511553385 +0000 UTC m=+5924.809834210" watchObservedRunningTime="2026-11-02 09:03:44.513501558 +0000 UTC m=+5924.811782432" Nov 02 09:03:44.820315 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:44.820280 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kk7sz_92ebc3a2-3bc6-48d7-b78c-f2dd729c4759/node-exporter/0.log" Nov 02 09:03:44.935062 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:44.935032 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kk7sz_92ebc3a2-3bc6-48d7-b78c-f2dd729c4759/kube-rbac-proxy/0.log" Nov 02 09:03:45.054773 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:45.054745 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kk7sz_92ebc3a2-3bc6-48d7-b78c-f2dd729c4759/init-textfile/0.log" Nov 02 09:03:45.558729 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:45.558689 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5zkrh_8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7/kube-rbac-proxy-main/0.log" Nov 02 09:03:45.676352 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:45.676312 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5zkrh_8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7/kube-rbac-proxy-self/0.log" Nov 02 09:03:45.805189 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:45.805138 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5zkrh_8fa67936-9e6b-4f8e-ae72-77f9a41f4dd7/openshift-state-metrics/0.log" Nov 02 09:03:45.946299 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:45.946219 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4602a0a9-fa2d-4233-b04b-0a33011f8021/prometheus/0.log" Nov 02 09:03:46.058807 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:46.058772 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4602a0a9-fa2d-4233-b04b-0a33011f8021/config-reloader/0.log" Nov 02 09:03:46.206490 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:46.206414 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4602a0a9-fa2d-4233-b04b-0a33011f8021/thanos-sidecar/0.log" Nov 02 09:03:46.331662 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:46.331632 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4602a0a9-fa2d-4233-b04b-0a33011f8021/kube-rbac-proxy-web/0.log" Nov 02 09:03:46.447272 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:46.447241 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4602a0a9-fa2d-4233-b04b-0a33011f8021/kube-rbac-proxy/0.log" Nov 02 09:03:46.637331 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:46.637298 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4602a0a9-fa2d-4233-b04b-0a33011f8021/kube-rbac-proxy-thanos/0.log" Nov 02 09:03:46.754272 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:46.754244 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4602a0a9-fa2d-4233-b04b-0a33011f8021/init-config-reloader/0.log" Nov 02 09:03:47.133822 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:47.133790 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-g8nbx_5ac39273-9863-4a60-86f4-29e64cbe2946/manager/0.log" Nov 02 09:03:47.248473 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:47.248437 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kv7wp_62de7c44-3318-4553-b3d0-f0124ed28ac4/manager/0.log" Nov 02 09:03:47.275845 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:47.275813 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7946767c4b-g78ln_f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8/telemeter-client/0.log" Nov 02 09:03:47.430985 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:47.430910 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7946767c4b-g78ln_f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8/reload/0.log" Nov 02 09:03:47.549269 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:47.549235 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7946767c4b-g78ln_f3301cf0-cecf-4a6a-a1b7-1be6d87b61c8/kube-rbac-proxy/0.log" Nov 02 09:03:47.683060 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:47.682983 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6499784568-r4gwp_2ebdbc81-5857-429e-9a08-2143afcdc097/thanos-query/0.log" Nov 02 09:03:47.835037 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:47.835004 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6499784568-r4gwp_2ebdbc81-5857-429e-9a08-2143afcdc097/kube-rbac-proxy-web/0.log" Nov 02 09:03:47.985108 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:47.985074 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6499784568-r4gwp_2ebdbc81-5857-429e-9a08-2143afcdc097/kube-rbac-proxy/0.log" Nov 02 09:03:48.019663 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:48.019622 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-kqv85_580bc81e-1b83-472a-9af3-153ba348eab7/manager/0.log" Nov 02 09:03:48.109553 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:48.109518 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6499784568-r4gwp_2ebdbc81-5857-429e-9a08-2143afcdc097/prom-label-proxy/0.log" Nov 02 09:03:48.238572 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:48.238499 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pq62z_787dc60b-c7be-4b20-9b11-daf3140a7589/server/0.log" Nov 02 09:03:48.238969 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:48.238597 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6499784568-r4gwp_2ebdbc81-5857-429e-9a08-2143afcdc097/kube-rbac-proxy-rules/0.log" Nov 02 09:03:48.354375 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:48.354342 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6499784568-r4gwp_2ebdbc81-5857-429e-9a08-2143afcdc097/kube-rbac-proxy-metrics/0.log" Nov 02 09:03:49.932217 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:49.932148 5899 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g4ck_d6e945a4-9c6d-4310-902f-eff8fc826621/konnectivity-agent/0.log" limit=1024 Nov 02 09:03:50.231139 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:50.231098 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-g8nbx_5ac39273-9863-4a60-86f4-29e64cbe2946/manager/0.log" Nov 02 09:03:50.348456 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:50.348412 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kv7wp_62de7c44-3318-4553-b3d0-f0124ed28ac4/manager/0.log" Nov 02 09:03:50.449495 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:50.449463 5899 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-98lt8" Nov 02 09:03:51.093812 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:51.093771 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-kqv85_580bc81e-1b83-472a-9af3-153ba348eab7/manager/0.log" Nov 02 09:03:51.333724 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:51.333699 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pq62z_787dc60b-c7be-4b20-9b11-daf3140a7589/server/0.log" Nov 02 09:03:53.378389 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:53.378354 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-g8nbx_5ac39273-9863-4a60-86f4-29e64cbe2946/manager/0.log" Nov 02 09:03:53.525110 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:53.525074 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kv7wp_62de7c44-3318-4553-b3d0-f0124ed28ac4/manager/0.log" Nov 02 09:03:54.243007 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:54.242965 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-kqv85_580bc81e-1b83-472a-9af3-153ba348eab7/manager/0.log" Nov 02 09:03:54.494520 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:54.494437 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pq62z_787dc60b-c7be-4b20-9b11-daf3140a7589/server/0.log" Nov 02 09:03:57.129072 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:57.129037 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kp755_1b808d2a-56c8-477f-bca0-d9d69d711516/dns/0.log" Nov 02 09:03:57.252017 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:57.251990 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kp755_1b808d2a-56c8-477f-bca0-d9d69d711516/kube-rbac-proxy/0.log" Nov 02 09:03:57.268959 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:57.268932 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6946f9c7-gkrcg_ea129fbd-3cf5-4889-a684-4c55c0c13c1b/kube-auth-proxy/0.log" Nov 02 09:03:57.472888 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:57.472815 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-7xqxx_e43d749b-223f-4a0a-997a-a0637ddf2d00/istio-proxy/0.log" Nov 02 09:03:57.512492 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:57.512458 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f6f9k_fdb3f2d5-5e21-48c3-af74-eb8fd530046a/dns-node-resolver/0.log" Nov 02 09:03:57.539552 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:57.539513 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-cf7zm_b0c571f9-9dc1-493e-b1e5-a256cad13493/istio-proxy/0.log" Nov 02 09:03:57.608745 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:57.608715 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-hkc5n_14f48878-440e-45ac-900a-a29238f348c9/istio-proxy/0.log" Nov 02 09:03:57.633745 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:57.633720 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6946f9c7-gkrcg_ea129fbd-3cf5-4889-a684-4c55c0c13c1b/kube-auth-proxy/0.log" Nov 02 09:03:57.859549 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:57.859514 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-7xqxx_e43d749b-223f-4a0a-997a-a0637ddf2d00/istio-proxy/0.log" Nov 02 09:03:57.923530 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:57.923498 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-cf7zm_b0c571f9-9dc1-493e-b1e5-a256cad13493/istio-proxy/0.log" Nov 02 09:03:57.988379 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:57.988346 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-hkc5n_14f48878-440e-45ac-900a-a29238f348c9/istio-proxy/0.log" Nov 02 09:03:58.936089 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:03:58.936053 5899 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-72ngx_e229a07c-7415-4af9-a9eb-9e7984e0a510/node-ca/0.log" Nov 02 09:04:02.016712 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:04:02.016671 5899 scope.go:122] "RemoveContainer" containerID="00bfe2499cc66c1614a4a930318e97fb6d344f249b610e2572fdeffa5b71f2de" Nov 02 09:04:02.026849 ip-10-0-135-6 kubenswrapper[5899]: I1102 09:04:02.026822 5899 scope.go:122] "RemoveContainer" containerID="39c459cf06f17b4c45d073f1cb482da15b4a1eea029879b728eb16d9aa09e1aa"